Graphical user interface for planning a procedure

ABSTRACT

A system and method of planning a procedure includes a planning workstation including: a display system; and a user input device. The planning workstation is configured to: display image data via the display system; receive a first user input via the user input device; display via the display system a target of a medical procedure within the displayed image data identified based at least on the first user input; display an interactive image via the display system, the interactive image including the image data, a plurality of connected anatomical passageways, and the identified target; receive a second user input via the user input device; display via the display system a trajectory between the target and an exit point along a nearest passageway of the plurality of connected anatomical passageways identified based at least on the second user input; receive a third user input via the user input device; and adjust the interactive image based at least on the defined trajectory and the third user input.

RELATED APPLICATIONS

The present disclosure claims priority to U.S. Provisional PatentApplication 62/486,896, filed Apr. 18, 2017, entitled “Graphical UserInterface for Planning a Procedure,” and U.S. Provisional PatentApplication 62/578,189, filed Oct. 27, 2017, entitled “Graphical UserInterface for Labeling Anatomy,” which are hereby incorporated byreference in their entirety.

TECHNICAL FIELD

The present disclosure is directed to systems and methods for performingan image-guided procedure and more particularly to systems and methodsfor analyzing, identifying, and/or labeling anatomy using a graphicaluser interface.

BACKGROUND

Minimally invasive medical techniques are intended to reduce the amountof tissue that is damaged during medical procedures, thereby reducingpatient recovery time, discomfort, and harmful side effects. Suchminimally invasive techniques may be performed through natural orificesin a patient anatomy or through one or more surgical incisions. Throughthese natural orifices or incisions clinicians may insert minimallyinvasive medical instruments (including surgical, diagnostic,therapeutic, or biopsy instruments) to reach a target tissue location.One such minimally invasive technique is to use a flexible elongatedevice, such as a catheter, which may be steerable, that can be insertedinto anatomic passageways and navigated toward a region of interestwithin the patient anatomy. Control of such an elongate device bymedical personnel during an image-guided procedure involves themanagement of several degrees of freedom including at least themanagement of insertion and retraction of the elongate device as well assteering or bend radius of the device. In addition, different modes ofoperation may also be supported.

Accordingly, it would be advantageous to provide a graphical userinterface that supports intuitive planning of medical proceduresincluding minimally invasive medical techniques.

SUMMARY

The embodiments of the invention are best summarized by the claims thatfollow the description.

According to some embodiments, a method for planning a medical procedureusing a graphical user interface may include displaying image data viathe graphical user interface and receiving a first user input defining atarget of the medical procedure within the displayed image data. Themethod may further include displaying an interactive image via thegraphical user interface, the interactive image comprising the imagedata, a plurality of connected anatomical passageways detected bysegmentation of the image data, and the defined target. The method mayfurther include receiving a second user input defining a trajectorybetween the target and an exit point along a nearest passageway of theplurality of connected anatomical passageways and receiving a third userinput adjusting the interactive image based on the defined trajectory.

According to some embodiments, a method for planning a medical procedureusing a graphical user interface may include displaying image data viathe graphical user interface, receiving a first user input defining ahazard within the displayed image data, and displaying an interactiveimage. The interactive image includes the image data, a plurality ofconnected anatomical passageways detected by segmentation of the imagedata, and the defined hazard.

According to some embodiments, a method for previewing a plan for amedical procedure using a graphical user interface may include providinga plurality of interactive windows for a user to view the plan for themedical procedure. Each of the plurality of interactive windows maydisplay a different rendering of a model of anatomical passageways. Themethod may further include displaying a path through the anatomicalpassageways to a target of the medical procedure, displaying a virtualimage of an instrument within the anatomical passageways, displaying acontrol point corresponding to a distal end of the instrument in atleast one of the plurality of interactive windows, receiving a userinput defining a position of the control point, and in response toreceiving the user input, dynamically updating a position of theinstrument in each of the plurality of interactive windows to match theposition of the control point.

According to some embodiments, a planning workstation may include adisplay system and a user input device. The planning workstation may beconfigured to display image data via the display system, receive a firstuser input via the user input device, the first user input defining atarget of the medical procedure within the displayed image data, displayan interactive image via the display system, the interactive imagecomprising the image data, a plurality of connected anatomicalpassageways detected by segmentation of the image data, and the definedtarget, receive a second user input via the user input device, thesecond user input defining a trajectory between the target and an exitpoint along a nearest passageway of the plurality of connectedanatomical passageways, and receive a third user input via the userinput device, the third user input adjusting the interactive image basedon the defined trajectory.

According to some embodiments, a non-transitory machine readable mediummay include a plurality of machine readable instructions which whenexecuted by one or more processors associated with a planningworkstation are adapted to cause the one or more processors to perform amethod. The method may include displaying image data via the graphicaluser interface, receiving a first user input defining a hazard withinthe displayed image data, and displaying an interactive image. Theinteractive image may comprise the image data, a plurality of connectedanatomical passageways detected by segmentation of the image data, andthe defined hazard.

According to some embodiments, a non-transitory machine readable mediummay include a plurality of machine readable instructions which whenexecuted by one or more processors associated with a planningworkstation are adapted to cause the one or more processors to perform amethod. The method may include providing a plurality of interactivewindows for a user to view the plan for the medical procedure,displaying a path through the anatomical passageways to a target of themedical procedure, displaying a virtual image of an instrument withinthe anatomical passageways, displaying a control point corresponding toa distal end of the instrument in at least one of the plurality ofinteractive windows, receiving a user input defining a position of thecontrol point, and in response to receiving the user input, dynamicallyupdating a position of the instrument in each of the plurality ofinteractive windows to match the position of the control point. Each ofthe plurality of interactive windows displays a different rendering of amodel of anatomical passageways.

According to some embodiments, a method of planning a medical proceduremay include receiving imaging data and rendering a model of anatomicalpassageways based on the imaging data, the anatomical passagewaysincluding a plurality of branches. An image of the model may bedisplayed via a graphical user interface. A first user inputrepresenting selection of a first label may be received, and a seconduser input representing selection of a first branch of the plurality ofbranches may be received. In response to the first user input and thesecond user input, the first branch may be labeled with the first label,and a representation of the first label applied to the first branch maybe displayed via the graphical user interface.

According to some embodiments, a non-transitory machine readable mediummay include a plurality of machine readable instructions. Theinstructions may cause the one or more processors to display, via agraphical user interface, a model of anatomical passageways including aplurality of branches; display, via the graphical user interface, a listof anatomical labels; receive a first user input that selects a firstlabel from the list of anatomical labels; receive a second user inputthat selects a first branch of the plurality of branches; and/or applythe first label to the first branch.

According to some embodiments, a planning workstation may include adisplay system and a user input device. The planning workstation may beconfigured to display anatomical passageways that include a plurality ofbranches, display a list of labels, receive a first user input via theuser input device selecting a first branch of the plurality of branches,receive a second user input via the user input device selecting a firstlabel from the list of labels, and in response to the first user inputand the second user input, display a representation of the first labelapplied to the first branch via the display system.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory innature and are intended to provide an understanding of the presentdisclosure without limiting the scope of the present disclosure. In thatregard, additional aspects, features, and advantages of the presentdisclosure will be apparent to one skilled in the art from the followingdetailed description.

BRIEF DESCRIPTIONS OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a simplified diagram of a teleoperated medical systemaccording to some embodiments.

FIG. 2A is a simplified diagram of a medical instrument system accordingto some embodiments.

FIG. 2B is a simplified diagram of a medical instrument with an extendedmedical tool according to some embodiments.

FIGS. 3A and 3B are simplified diagrams of side views of a patientcoordinate space including a medical instrument mounted on an insertionassembly according to some embodiments.

FIG. 4 is a simplified diagram of a graphical user interface in a dataselection mode according to some embodiments.

FIGS. 5A-5G are simplified diagrams of a graphical user interface in ahybrid segmentation and planning mode according to some embodiments

FIG. 6 is a simplified diagram of a graphical user interface in apreview mode according to some embodiments.

FIG. 7 is a simplified diagram of a graphical user interface in a savemode according to some embodiments.

FIG. 8 is a simplified diagram of a graphical user interface in amanagement mode according to some embodiments.

FIG. 9 is a simplified diagram of a graphical user interface in a reviewmode according to some embodiments.

FIG. 10 is a simplified diagram of a method for planning a medicalprocedure according to some embodiments.

FIG. 11 is a simplified diagram of a method for modifying an anatomicalrepresentation (e.g., a model) to provide access to a target of amedical procedure according to some embodiments.

FIG. 12 is a simplified diagram of a method for growing an anatomicalrepresentation (e.g., a model) to provide access to a target of amedical procedure according to some embodiments.

FIG. 13 is a simplified diagram of a method for planning a medicalprocedure using a graphical user interface according to someembodiments.

FIGS. 14A-14F are simplified diagrams of a graphical user interface in abranch labeling mode according to some embodiments.

FIGS. 15A and 15B are simplified diagrams of a method of applying alabel to a branched model of anatomical passageways according to someembodiments.

FIG. 16 is a simplified diagram of a method for planning a medicalprocedure according to some embodiments.

FIGS. 17A-17N are diagrams of graphical user interface corresponding tothe performance of the method for planning a medical procedure accordingto some embodiments

Embodiments of the present disclosure and their advantages are bestunderstood by referring to the detailed description that follows. Itshould be appreciated that like reference numerals are used to identifylike elements illustrated in one or more of the figures, whereinshowings therein are for purposes of illustrating embodiments of thepresent disclosure and not for purposes of limiting the same.

DETAILED DESCRIPTION

In the following description, specific details are set forth describingsome embodiments consistent with the present disclosure. Numerousspecific details are set forth in order to provide a thoroughunderstanding of the embodiments. It will be apparent, however, to oneskilled in the art that some embodiments may be practiced without someor all of these specific details. The specific embodiments disclosedherein are meant to be illustrative but not limiting. One skilled in theart may realize other elements that, although not specifically describedhere, are within the scope and the spirit of this disclosure. Inaddition, to avoid unnecessary repetition, one or more features shownand described in association with one embodiment may be incorporatedinto other embodiments unless specifically described otherwise or if theone or more features would make an embodiment non-functional.

In some instances well known methods, procedures, components, andcircuits have not been described in detail so as not to unnecessarilyobscure aspects of the embodiments.

This disclosure describes various instruments and portions ofinstruments in terms of their state in three-dimensional space. As usedherein, the term “position” refers to the location of an object or aportion of an object in a three-dimensional space (e.g., three degreesof translational freedom along Cartesian x-, y-, and z-coordinates). Asused herein, the term “orientation” refers to the rotational placementof an object or a portion of an object (three degrees of rotationalfreedom—e.g., roll, pitch, and yaw). As used herein, the term “pose”refers to the position of an object or a portion of an object in atleast one degree of translational freedom and to the orientation of thatobject or portion of the object in at least one degree of rotationalfreedom (up to six total degrees of freedom). As used herein, the term“shape” refers to a set of poses, positions, or orientations measuredalong an object.

One general aspect of the present disclosure includes a method forplanning a medical procedure, the method including: displaying imagedata via a graphical user interface; receiving, by the graphical userinterface, a first user input; identifying at least a portion of atarget within the displayed image data using the first user input;displaying an interactive image via the graphical user interface, theinteractive image including the image data, a plurality of connectedanatomical passageways associated with the image data, and theidentified target; receiving a second user input; identifying at least aportion of a trajectory between the target and an exit point along anearest connected passageway of the plurality of connected anatomicalpassageways using the second user input; receiving a third user input;and adjusting the interactive image based at least on the identifiedtrajectory and using the third user input. Implementations may includeone or more of the following features. The method including providing aline tool via the graphical user interface to receive the second userinput. The method where adjusting the interactive image includes:determining a distance represented by the trajectory; determiningwhether the distance is greater than a predetermined threshold;receiving a fourth user input; identifying at least a portion of anunconnected passageway that is closer to the target than the nearestconnected passageway using the fourth user input; and connecting theunconnected passageway to the plurality of connected passageways. Themethod where identifying the unconnected passageway includes receiving afifth user input and iteratively rotating the interactive image toidentify the unconnected passageway in the interactive image using thefifth user input. The method where the interactive image is iterativelyrotated about one or more user-defined rotation points. The methodfurther including identifying an axis of rotation based at least on theone or more user-defined rotation points. The method where theinteractive image is iteratively rotated about the axis of rotation. Themethod where adjusting the interactive image includes: determining anexit angle based on the trajectory; and adjusting the exit angle byaltering a position of the exit point along the nearest connectedpassageway. The method including providing a slider via the graphicaluser interface and receiving user input via the slider that alters theexit point. The method further including receiving a fourth user inputand identifying a hazard of the medical procedure within the displayedimage data using the sixth user input. The method where the hazardcorresponds to a vulnerable portion of a patient anatomy. The methodwhere the hazard corresponds to an excessive bend in one or more of theplurality of connected anatomical passageways. The method includingdisplaying a hazard fence to represent the hazard. The method where thehazard fence includes at least one of a circular disk, a conic hazardfence, and a hemispherical hazard fence. The method further includingreceiving a fourth user input and identifying at least a portion of apath within the plurality of connected passageways to the target usingthe seventh user input. The method where the first user input isreceived before completing segmentation of the image date. The methodwhere displaying the interactive image includes overlaying the pluralityof connected anatomical passageways on the displayed image data, andwhere the plurality of connected anatomical passageways are dynamicallyupdated to reflect progress of the segmentation of the image data. Themethod further including receiving a fourth user input and identifyingat least a portion of a passageway among the plurality of connectedanatomical passageways to be disconnected from the plurality ofconnected anatomical passageways. The method where the plurality ofconnected anatomical passageways include lung airways.

One general aspect of the present disclosure includes a method forplanning a medical procedure, the method including: providing agraphical user interface; displaying image data via the graphical userinterface; receiving a first user input; identifying at least a portionof a hazard within the displayed image data using the first user input;and displaying an interactive image including the image data, aplurality of connected anatomical passageways, and the a representationof the identified hazard. Implementations may include one or more of thefollowing features. The method where the hazard includes a vulnerableportion of a patient anatomy. The method where the hazard includes anexcessive bend within the plurality of connected anatomical passageways.The method including displaying a hazard fence to represent the hazard.The method where the hazard fence includes at least one of a circulardisk, a conical hazard fence, and a hemispherical hazard fence. Themethod where the plurality of connected anatomical passageways includeslung airways. The method further including receiving a second userinput, identifying at least a portion of a target of the medicalprocedure within the displayed image data, and where the hazardcorresponds to a vulnerable portion of a patient anatomy that is closeto the target. The method where the vulnerable portion of the patientanatomy includes at least one of a lung pleura, a blood vessel, largebullae, and a heart. The method where the first user input is receivedbefore the segmentation of the image data is complete.

One general aspect of the present disclosure includes a method forpreviewing a plan for a medical procedure, the method including:providing a graphical user interface including a plurality ofinteractive windows that display the plan for the medical procedure,where at least two different renderings of a model of anatomicalpassageways are displayed using the plurality of interactive windows;displaying a path through the anatomical passageways to a target of themedical procedure; displaying a virtual image of an instrument withinthe anatomical passageways; displaying a control point corresponding toa distal end of the instrument in at least one of the plurality ofinteractive windows; receiving a user input; identifying a position ofthe control point using the user input; and in response to receiving theuser input, dynamically updating a position of the instrument in atleast two of the plurality of interactive windows to match the positionof the control point.

One general aspect of the present disclosure includes a planningworkstation including: a display system; and a user input device; wherethe planning workstation is configured to: display image data via thedisplay system; receive a first user input via the user input device;display via the display system a target of a medical procedure withinthe displayed image data identified based at least on the first userinput; display an interactive image via the display system, theinteractive image including the image data, a plurality of connectedanatomical passageways, and the identified target; receive a second userinput via the user input device; display via the display system atrajectory between the target and an exit point along a nearestpassageway of the plurality of connected anatomical passagewaysidentified based at least on the second user input; receive a third userinput via the user input device; and adjust the interactive image basedat least on the defined trajectory and the third user input.Implementations may include one or more of the following features. Theplanning workstation where the user input device includes a touchscreenof the display system. The planning workstation where the display systemadjusts the interactive image including: determine a distancerepresented by the trajectory; determine whether the distance is greaterthan a predetermined threshold; receive a fourth user input via the userinput; identify an unconnected passageway that is closer to the targetthan the nearest connected passageway based at least on the fourth userinput; and connect the unconnected passageway to the plurality ofconnected passageways. The planning workstation where the planningworkstation is further configured to receive a fifth user input via theuser input device, and rotate the interactive image to identify theunconnected passageway in the interactive image based at least on thefifth user input. The planning workstation where the interactive imageis rotated about one or more user-defined rotation points. The planningworkstation where the planning workstation is further configured toidentify an axis of rotation based on the one or more user-definedrotation points. The planning workstation where the display systemadjusts the interactive image including: determine an exit angle basedon the trajectory; and adjust the exit angle by altering a position ofthe exit point along the nearest connected passageway. The planningworkstation where the planning workstation is further configured toreceive a fourth user input via the user input device, and display viathe display system a hazard of the medical procedure within thedisplayed image data based on the fourth user input. The planningworkstation where the hazard corresponds to at least one of a vulnerableportion of a patient anatomy and an excessive bend in one or more of theplurality of connected anatomical passageways. The planning workstationwhere the hazard is displayed using a hazard fence that includes atleast one of a circular disk, a conic hazard fence, and a hemisphericalhazard fence. The planning workstation where the user input device isconfigured to receive first user before segmentation of the image dateis complete.

One general aspect of the present disclosure includes a non-transitorymachine readable medium including a plurality of machine readableinstructions which when executed by one or more processors associatedwith a planning workstation are adapted to cause the one or moreprocessors to perform a method including: displaying image data via agraphical user interface; receiving a first user input; identifying ahazard within the displayed image data based at least on the first userinput; and displaying an interactive image including the image data, aplurality of connected anatomical passageways detected by segmentationof the image data, and the identified hazard. Implementations mayinclude one or more of the following features. The non-transitorymachine readable medium where the hazard includes a vulnerable portionof a patient anatomy. The non-transitory machine readable medium wherethe hazard includes an excessive bend within the plurality of connectedanatomical passageways. The non-transitory machine readable medium wherethe hazard is represented using a hazard fence. The non-transitorymachine readable medium where the hazard fence includes at least one ofa circular disk, a conical hazard fence, and a hemispherical hazardfence. The non-transitory machine readable medium where the plurality ofconnected anatomical passageways includes lung airways. Thenon-transitory machine readable medium where the machine readableinstructions are adapted to cause the one or more processors to performthe method further including receiving a second user input, identifyinga target of a medical procedure within the displayed image data using atleast the second user input, and where the hazard corresponds to avulnerable portion of a patient anatomy that is close to the target. Thenon-transitory machine readable medium where the vulnerable portion ofthe patient anatomy includes at least one of a lung pleura, a bloodvessel, large bullae, and a heart. The non-transitory machine readablemedium where the machine readable instructions are adapted to cause theone or more processors to perform the method including receiving thefirst user input segmentation of the image data is complete.

One general aspect of the present disclosure includes a non-transitorymachine readable medium including a plurality of machine readableinstructions which when executed by one or more processors associatedwith a planning workstation are adapted to cause the one or moreprocessors to perform a method including: providing a plurality ofinteractive windows for a user to view the plan for a medical procedure,where each of the plurality of interactive windows displays a differentrendering of a model of anatomical passageways; displaying a paththrough the anatomical passageways to a target of the medical procedure;displaying a virtual image of an instrument within the anatomicalpassageways; displaying a control point corresponding to a distal end ofthe instrument in at least one of the plurality of interactive windows;receiving a user input; identifying a position of the control pointbased at least on the user input; and in response to receiving the userinput, dynamically updating a position of the instrument in at least oneof the plurality of interactive windows to match the position of thecontrol point.

One general aspect of the present disclosure includes a method ofplanning a medical procedure, the method including: receiving arepresentation of anatomical passageways including a plurality ofbranches; displaying, via a graphical user interface, an image of therepresentation; receiving a first user input representing selection of afirst label; receiving a second user input representing selection of afirst branch of the plurality of branches; and in response to the firstuser input and the second user input: labeling the first branch with thefirst label; and displaying, via the graphical user interface, arepresentation of the first label applied to the first branch.Implementations may include one or more of the following features. Themethod further including, based on the labeling of the first branch withthe first label: selecting a second label; and displaying, via thegraphical user interface, an indication that the second label has beenselected. The method where the second label is selected based on anarrangement of the plurality of branches within the anatomicalpassageways. The method further including: receiving a third user inputrepresenting selection of a second branch of the plurality of branches;labeling the second branch with the second label; and displaying, viathe graphical user interface, a representation of the second labelapplied to the second branch. The method further including: identifyinga group of branches from the plurality of branches that includes thefirst branch; and in response to the first user input and the seconduser input, labeling the group of branches with the first label. Themethod where the identifying of the group of branches includesidentifying a descendent branch of the first branch and including thedescendent branch in the group of branches. The method where theidentifying of the group of branches includes identifying an antecedentdescendent branch of the first branch and including the antecedentbranch in the group of branches. The method further including,displaying via the graphical user interface, an indication of a secondbranch of the plurality of branches that does not have an assignedlabel. The method further including: receiving a third user inputrepresenting selection of a second label; labeling the second branchwith the second label; and displaying, via the graphical user interface,a representation of the second label applied to the second branch. Themethod where the representation of the anatomical passageways is basedon imaging data of a patient. The method where the imaging data includesan anatomical structure and the displaying of the image of therepresentation of the anatomical passageways displays the anatomicalpassageways and the anatomical structure. The method further including,in response to the first user input and the second user input: assigninga color to the first branch; and displaying the image of therepresentation of anatomical passageways with the first branch coloredwith the assigned color. The method further including: providing, viathe graphical user interface, a cursor; detecting that the cursor isaligned with the first branch of the plurality of branches; and based ondetecting that the cursor is aligned with the first branch, modifying arepresentation of the cursor. The method further including: receiving athird user input representing a rotation instruction; in response to thethird user input, rotating the representation of anatomical passageways;and displaying, via the graphical user interface, an image of therotated representation of anatomical passageways.

One general aspect of the present disclosure includes a non-transitorymachine readable medium including a plurality of machine readableinstructions, which when executed by one or more processors, cause theone or more processors to perform operations including: display, via agraphical user interface, a representation of anatomical passageways,where the anatomical passageways include a plurality of branches;display, via the graphical user interface, a list of anatomical labels;receive a first user input that selects a first label from the list ofanatomical labels; receive a second user input that selects a firstbranch of the plurality of branches; and apply the first label to thefirst branch. Implementations may include one or more of the followingfeatures. The non-transitory machine readable medium where the firstbranch is included in a group of branches, the non-transitory machinereadable medium including further instructions that cause the one ormore processors to, based on the second user input that selects thefirst branch of the plurality of branches, apply the first label to thegroup of branches. The non-transitory machine readable medium includingfurther instructions that cause the one or more processors to identifythe group of branches by identifying a descendant branch of the firstbranch and adding the descendent branch to the group of branches. Thenon-transitory machine readable medium including further instructionsthat cause the one or more processors to perform operations including:upon applying the first label to the first branch, select a second labelbased on an arrangement of branches within the anatomical passageways.The non-transitory machine readable medium including furtherinstructions that cause the one or more processors to perform operationsincluding: receive a third user input that selects a second branch ofthe plurality of branches; and apply the second label to the secondbranch. The non-transitory machine readable medium including furtherinstructions that cause the one or more processors to perform operationsincluding: upon applying each label in the list of anatomical labels:identify a second branch of the plurality of branches that is unlabeled;and display, via the graphical user interface, an indication that thesecond branch is unlabeled. The non-transitory machine readable mediumincluding further instructions that cause the one or more processors toperform operations including: receive a third user input that selects asecond label from the list of anatomical labels; and apply the secondlabel to the second branch. The non-transitory machine readable mediumincluding further instructions that cause the one or more processors toperform operations including: display, via the graphical user interface,a status indicator for the second label that indicates that the secondlabel is assigned to more than one branch. The non-transitory machinereadable medium including further instructions that cause the one ormore processors to perform operations including: display, via thegraphical user interface, a representation of the first label beingapplied to the first branch. The non-transitory machine readable mediumincluding further instructions that cause the one or more processors to,based on applying the first label to the first branch, performoperations including: assign a color to the first branch; and display,via the graphical user interface, the representation of the anatomicalpassageways with the first branch colored the assigned color. Thenon-transitory machine readable medium where the first branch isincluded in a group of branches, the non-transitory machine readablemedium including further instructions that cause the one or moreprocessors to, based on applying the first label to the first branch,perform operations including: assign the color to the group of branches;and display, via the graphical user interface, the representation of theanatomical passageways with the group of branches colored the assignedcolor. The non-transitory machine readable medium including furtherinstructions that cause the one or more processors to: provide, via thegraphical user interface, a cursor; detect that the cursor is alignedwith one of the plurality of branches; and based on detecting that thecursor is aligned, modify a representation of the cursor. Thenon-transitory machine readable medium including further instructionsthat cause the one or more processors to: based on the first user inputthat selects the first label and the second user input that selects thefirst branch of the plurality of branches, compare the first label andthe first branch to a second label applied to a second branch todetermine whether the first label and the second label conflict; andwhen determined that the first label and the second label do notconflict, apply the first label to the first branch.

One general aspect of the present disclosure includes a planningworkstation including: a display system; and a user input device; wherethe planning workstation is configured to: display anatomicalpassageways that include a plurality of branches via the display system;display a list of labels via the display system; receive a first userinput via the user input device selecting a first branch of theplurality of branches; receive a second user input via the user inputdevice selecting a first label from the list of labels; and in responseto the first user input and the second user input, display arepresentation of the first label applied to the first branch via thedisplay system. Implementations may include one or more of the followingfeatures. The planning workstation where the planning workstation isfurther configured to, in response to the first label being applied tothe first branch, select a second label from the list of labels based onan arrangement of branches within the anatomical passageways. Theplanning workstation where the planning workstation is furtherconfigured to perform operations including: identify a second branch ofthe plurality of branches that is unlabeled; and display an indicationthat the second branch is unlabeled via the display system. The planningworkstation where the planning workstation is further configured to, inresponse to the first user input selecting the first branch, performoperations including: identify a group of branches from the plurality ofbranches that includes the first branch; apply the first label to thegroup of branches; and display a representation of the first labelapplied to the group of branches via the display system. The planningworkstation where the planning workstation is further configured to, inresponse to the first user input selecting the first branch, performoperations including: determine whether the first label conflicts with asecond label; and when determined that the first label does not conflictwith the second label, apply the first label to the first branch.

FIG. 1 is a simplified diagram of a teleoperated medical system 100according to some embodiments. In some embodiments, teleoperated medicalsystem 100 may be suitable for use in, for example, surgical,diagnostic, therapeutic, or biopsy procedures. As shown in FIG. 1,medical system 100 generally includes a teleoperational manipulatorassembly 102 for operating a medical instrument 104 in performingvarious procedures on a patient P. Teleoperational manipulator assembly102 is mounted to or near an operating table T. A master assembly 106allows an operator (e.g., a surgeon, a clinician, or a physician O asillustrated in FIG. 1) to view the interventional site and to controlteleoperational manipulator assembly 102.

Master assembly 106 may be located at a surgeon's console which isusually located in the same room as operating table T, such as at theside of a surgical table on which patient P is located. However, itshould be understood that physician O can be located in a different roomor a completely different building from patient P. Master assembly 106generally includes one or more control devices for controllingteleoperational manipulator assembly 102. The control devices mayinclude any number of a variety of input devices, such as joysticks,trackballs, data gloves, trigger-guns, hand-operated controllers, voicerecognition devices, body motion or presence sensors, and/or the like.To provide physician O a strong sense of directly controllinginstruments 104 the control devices may be provided with the samedegrees of freedom as the associated medical instrument 104. In thismanner, the control devices provide physician O with telepresence or theperception that the control devices are integral with medicalinstruments 104.

In some embodiments, the control devices may have more or fewer degreesof freedom than the associated medical instrument 104 and still providephysician O with telepresence. In some embodiments, the control devicesmay optionally be manual input devices which move with six degrees offreedom, and which may also include an actuatable handle for actuatinginstruments (for example, for closing grasping jaws, applying anelectrical potential to an electrode, delivering a medicinal treatment,and/or the like).

Teleoperational manipulator assembly 102 supports medical instrument 104and may include a kinematic structure of one or more non-servocontrolled links (e.g., one or more links that may be manuallypositioned and locked in place, generally referred to as a set-upstructure) and a teleoperational manipulator. Teleoperationalmanipulator assembly 102 may optionally include a plurality of actuatorsor motors that drive inputs on medical instrument 104 in response tocommands from the control system (e.g., a control system 112). Theactuators may optionally include drive systems that when coupled tomedical instrument 104 may advance medical instrument 104 into anaturally or surgically created anatomic orifice. Other drive systemsmay move the distal end of medical instrument 104 in multiple degrees offreedom, which may include three degrees of linear motion (e.g., linearmotion along the X, Y, Z Cartesian axes) and in three degrees ofrotational motion (e.g., rotation about the X, Y, Z Cartesian axes).Additionally, the actuators can be used to actuate an articulable endeffector of medical instrument 104 for grasping tissue in the jaws of abiopsy device and/or the like. Actuator position sensors such asresolvers, encoders, potentiometers, and other mechanisms may providesensor data to medical system 100 describing the rotation andorientation of the motor shafts. This position sensor data may be usedto determine motion of the objects manipulated by the actuators.

Teleoperated medical system 100 may include a sensor system 108 with oneor more sub-systems for receiving information about the instruments ofteleoperational manipulator assembly 102. Such sub-systems may include aposition/location sensor system (e.g., an electromagnetic (EM) sensorsystem); a shape sensor system for determining the position,orientation, speed, velocity, pose, and/or shape of a distal end and/orof one or more segments along a flexible body that may make up medicalinstrument 104; and/or a visualization system for capturing images fromthe distal end of medical instrument 104.

Teleoperated medical system 100 also includes a display system 110 fordisplaying an image or representation of the surgical site and medicalinstrument 104 generated by sub-systems of sensor system 108. Displaysystem 110 and master assembly 106 may be oriented so physician O cancontrol medical instrument 104 and master assembly 106 with theperception of telepresence.

In some embodiments, medical instrument 104 may have a visualizationsystem (discussed in more detail below), which may include a viewingscope assembly that records a concurrent or real-time image of asurgical site and provides the image to the operator or physician Othrough one or more displays of medical system 100, such as one or moredisplays of display system 110. The concurrent image may be, forexample, a two or three dimensional image captured by an endoscopepositioned within the surgical site. In some embodiments, thevisualization system includes endoscopic components that may beintegrally or removably coupled to medical instrument 104. However insome embodiments, a separate endoscope, attached to a separatemanipulator assembly may be used with medical instrument 104 to imagethe surgical site. The visualization system may be implemented ashardware, firmware, software or a combination thereof which interactwith or are otherwise executed by one or more computer processors, whichmay include the processors of a control system 112.

Display system 110 may also display an image of the surgical site andmedical instruments captured by the visualization system. In someexamples, teleoperated medical system 100 may configure medicalinstrument 104 and controls of master assembly 106 such that therelative positions of the medical instruments are similar to therelative positions of the eyes and hands of physician O. In this mannerphysician O can manipulate medical instrument 104 and the hand controlas if viewing the workspace in substantially true presence. By truepresence, it is meant that the presentation of an image is a trueperspective image simulating the viewpoint of a physician that isphysically manipulating medical instrument 104.

In some examples, display system 110 may present images of a surgicalsite recorded pre-operatively or intra-operatively using image data fromimaging technology such as, computed tomography (CT), magnetic resonanceimaging (MM), fluoroscopy, thermography, ultrasound, optical coherencetomography (OCT), thermal imaging, impedance imaging, laser imaging,nanotube X-ray imaging, and/or the like. The pre-operative orintra-operative image data may be presented as two-dimensional,three-dimensional, or four-dimensional (including e.g., time based orvelocity based information) images and/or as images fromrepresentations, such as models created from the pre-operative orintra-operative image data sets.

In some embodiments, often for purposes of imaged guided surgicalprocedures, display system 110 may display a virtual navigational imagein which the actual location of medical instrument 104 is registered(i.e., dynamically referenced) with the preoperative or concurrentimages or representations (e.g., models). This may be done to presentthe physician O with a virtual image of the internal surgical site froma viewpoint of medical instrument 104. In some examples, the viewpointmay be from a tip of medical instrument 104. An image of the tip ofmedical instrument 104 and/or other graphical or alphanumeric indicatorsmay be superimposed on the virtual image to assist physician Ocontrolling medical instrument 104. In some examples, medical instrument104 may not be visible in the virtual image.

In some embodiments, display system 110 may display a virtualnavigational image in which the actual location of medical instrument104 is registered with preoperative or concurrent images to present thephysician O with a virtual image of medical instrument 104 within thesurgical site from an external viewpoint. An image of a portion ofmedical instrument 104 or other graphical or alphanumeric indicators maybe superimposed on the virtual image to assist physician O in thecontrol of medical instrument 104. As described herein, visualrepresentations of data points may be rendered to display system 110.For example, measured data points, moved data points, registered datapoints, and other data points described herein may be displayed ondisplay system 110 in a visual representation. The data points may bevisually represented in a user interface by a plurality of points ordots on display system 110 or as a rendered representation (e.g., arendered model), such as a mesh or wire model created based on the setof data points. In some examples, the data points may be color codedaccording to the data they represent. In some embodiments, a visualrepresentation may be refreshed in display system 110 after eachprocessing operation has been implemented to alter data points.

Teleoperated medical system 100 may also include control system 112.Control system 112 includes at least one memory and at least onecomputer processor (not shown) for effecting control between medicalinstrument 104, master assembly 106, sensor system 108, and displaysystem 110. Control system 112 also includes programmed instructions(e.g., a non-transitory machine-readable medium storing theinstructions) to implement some or all of the methods described inaccordance with aspects disclosed herein, including instructions forproviding information to display system 110. While control system 112 isshown as a single block in the simplified schematic of FIG. 1, thesystem may include two or more data processing circuits with one portionof the processing optionally being performed on or adjacent toteleoperational manipulator assembly 102, another portion of theprocessing being performed at master assembly 106, and/or the like. Theprocessors of control system 112 may execute instructions comprisinginstruction corresponding to processes disclosed herein and described inmore detail below. Any of a wide variety of centralized or distributeddata processing architectures may be employed. Similarly, the programmedinstructions may be implemented as a number of separate programs orsubroutines, or they may be integrated into a number of other aspects ofthe teleoperational systems described herein. In one embodiment, controlsystem 112 supports wireless communication protocols such as Bluetooth,IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.

In some embodiments, control system 112 may receive force and/or torquefeedback from medical instrument 104. Responsive to the feedback,control system 112 may transmit signals to master assembly 106. In someexamples, control system 112 may transmit signals instructing one ormore actuators of teleoperational manipulator assembly 102 to movemedical instrument 104. Medical instrument 104 may extend into aninternal surgical site within the body of patient P via openings in thebody of patient P. Any suitable conventional and/or specializedactuators may be used. In some examples, the one or more actuators maybe separate from, or integrated with, teleoperational manipulatorassembly 102. In some embodiments, the one or more actuators andteleoperational manipulator assembly 102 are provided as part of ateleoperational cart positioned adjacent to patient P and operatingtable T.

Control system 112 may optionally further include a virtualvisualization system to provide navigation assistance to physician Owhen controlling medical instrument 104 during an image-guided surgicalprocedure. Virtual navigation using the virtual visualization system maybe based upon reference to an acquired preoperative or intraoperativedataset of anatomic passageways. The virtual visualization systemprocesses images of the surgical site imaged using imaging technologysuch as computerized tomography (CT), magnetic resonance imaging (MRI),fluoroscopy, thermography, ultrasound, optical coherence tomography(OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-rayimaging, and/or the like. Software, which may be used in combinationwith manual inputs, is used to convert the recorded images intosegmented two dimensional or three dimensional composite representationof a partial or an entire anatomic organ or anatomic region. An imagedata set is associated with the composite representation. The compositerepresentation and the image data set describe the various locations andshapes of the passageways and their connectivity. The images used togenerate the composite representation may be recorded preoperatively orintra-operatively during a clinical procedure. In some embodiments, avirtual visualization system may use standard representations (i.e., notpatient specific) or hybrids of a standard representation and patientspecific data. The composite representation and any virtual imagesgenerated by the composite representation may represent the staticposture of a deformable anatomic region during one or more phases ofmotion (e.g., during an inspiration/expiration cycle of a lung).

During a virtual navigation procedure, sensor system 108 may be used tocompute an approximate location of medical instrument 104 with respectto the anatomy of patient P. The location can be used to produce bothmacro-level (external) tracking images of the anatomy of patient P andvirtual internal images of the anatomy of patient P. The system mayimplement one or more electromagnetic (EM) sensor, fiber optic sensors,and/or other sensors to register and display a medical implementtogether with preoperatively recorded surgical images. For example U.S.patent application Ser. No. 13/107,562 (filed May 13, 2011) (disclosing“Medical System Providing Dynamic Registration of a Model of an AnatomicStructure for Image-Guided Surgery”) which is incorporated by referenceherein in its entirety, discloses one such system. Teleoperated medicalsystem 100 may further include optional operations and support systems(not shown) such as illumination systems, steering control systems,irrigation systems, and/or suction systems. In some embodiments,teleoperated medical system 100 may include more than oneteleoperational manipulator assembly and/or more than one masterassembly. The exact number of teleoperational manipulator assemblieswill depend on the surgical procedure and the space constraints withinthe operating room, among other factors. Master assembly 106 may becollocated or they may be positioned in separate locations. Multiplemaster assemblies allow more than one operator to control one or moreteleoperational manipulator assemblies in various combinations.

FIG. 2A is a simplified diagram of a medical instrument system 200according to some embodiments. In some embodiments, medical instrumentsystem 200 may be used as medical instrument 104 in an image-guidedmedical procedure performed with teleoperated medical system 100. Insome examples, medical instrument system 200 may be used fornon-teleoperational exploratory procedures or in procedures involvingtraditional manually operated medical instruments, such as endoscopy.Optionally medical instrument system 200 may be used to gather (i.e.,measure) a set of data points corresponding to locations within anatomicpassageways of a patient, such as patient P.

Medical instrument system 200 includes elongate device 202 coupled to adrive unit 204. Elongate device 202 includes a flexible body 216 havingproximal end 217 and distal end or tip portion 218. In some embodiments,flexible body 216 has an approximately 3 mm outer diameter. Otherflexible body outer diameters may be larger or smaller.

Medical instrument system 200 further includes a tracking system 230 fordetermining the position, orientation, speed, velocity, pose, and/orshape of flexible body 216 at distal end 218 and/or of one or moresegments 224 along flexible body 216 using one or more sensors and/orimaging devices as described in further detail below. The entire lengthof flexible body 216, between distal end 218 and proximal end 217, maybe effectively divided into segments 224. If medical instrument system200 is consistent with medical instrument 104 of a teleoperated medicalsystem 100, tracking system 230. Tracking system 230 may optionally beimplemented as hardware, firmware, software or a combination thereofwhich interact with or are otherwise executed by one or more computerprocessors, which may include the processors of control system 112 inFIG. 1.

Tracking system 230 may optionally track distal end 218 and/or one ormore of the segments 224 using a shape sensor 222. Shape sensor 222 mayoptionally include an optical fiber aligned with flexible body 216(e.g., provided within an interior channel (not shown) or mountedexternally). In one embodiment, the optical fiber has a diameter ofapproximately 200 μm. In other embodiments, the dimensions may be largeror smaller. The optical fiber of shape sensor 222 forms a fiber opticbend sensor for determining the shape of flexible body 216. In onealternative, optical fibers including Fiber Bragg Gratings (FBGs) areused to provide strain measurements in structures in one or moredimensions. Various systems and methods for monitoring the shape andrelative position of an optical fiber in three dimensions are describedin U.S. patent application Ser. No. 11/180,389 (filed Jul. 13, 2005)(disclosing “Fiber optic position and shape sensing device and methodrelating thereto”); U.S. patent application Ser. No. 12/047,056 (filedon Jul. 16, 2004) (disclosing “Fiber-optic shape and relative positionsensing”); and U.S. Pat. No. 6,389,187 (filed on Jun. 17, 1998)(disclosing “Optical Fibre Bend Sensor”), which are all incorporated byreference herein in their entireties. Sensors in some embodiments mayemploy other suitable strain sensing techniques, such as Rayleighscattering, Raman scattering, Brillouin scattering, and Fluorescencescattering. In some embodiments, the shape of flexible body 216 may bedetermined using other techniques. For example, a history of the distalend pose of flexible body 216 can be used to reconstruct the shape offlexible body 216 over the interval of time. In some embodiments,tracking system 230 may optionally and/or additionally track distal end218 using a position sensor system 220. Position sensor system 220 maybe a component of an EM sensor system with positional sensor system 220including one or more conductive coils that may be subjected to anexternally generated electromagnetic field. Each coil of EM sensorsystem 220 then produces an induced electrical signal havingcharacteristics that depend on the position and orientation of the coilrelative to the externally generated electromagnetic field. In someembodiments, position sensor system 220 may be configured and positionedto measure six degrees of freedom, e.g., three position coordinates X,Y, Z and three orientation angles indicating pitch, yaw, and roll of abase point or five degrees of freedom, e.g., three position coordinatesX, Y, Z and two orientation angles indicating pitch and yaw of a basepoint. Further description of a position sensor system is provided inU.S. Pat. No. 6,380,732 (filed Aug. 11, 1999) (disclosing “Six-Degree ofFreedom Tracking System Having a Passive Transponder on the Object BeingTracked”), which is incorporated by reference herein in its entirety.

In some embodiments, tracking system 230 may alternately and/oradditionally rely on historical pose, position, or orientation datastored for a known point of an instrument system along a cycle ofalternating motion, such as breathing. This stored data may be used todevelop shape information about flexible body 216. In some examples, aseries of positional sensors (not shown), such as electromagnetic (EM)sensors similar to the sensors in position sensor 220 may be positionedalong flexible body 216 and then used for shape sensing. In someexamples, a history of data from one or more of these sensors takenduring a procedure may be used to represent the shape of elongate device202, particularly if an anatomic passageway is generally static.

Flexible body 216 includes a channel 221 sized and shaped to receive amedical instrument 226. FIG. 2B is a simplified diagram of flexible body216 with medical instrument 226 extended according to some embodiments.In some embodiments, medical instrument 226 may be used for proceduressuch as surgery, biopsy, ablation, illumination, irrigation, or suction.Medical instrument 226 can be deployed through channel 221 of flexiblebody 216 and used at a target location within the anatomy. Medicalinstrument 226 may include, for example, image capture probes, biopsyinstruments, laser ablation fibers, and/or other surgical, diagnostic,or therapeutic tools. Medical tools may include end effectors having asingle working member such as a scalpel, a blunt blade, an opticalfiber, an electrode, and/or the like. Other end effectors may include,for example, forceps, graspers, scissors, clip appliers, and/or thelike. Other end effectors may further include electrically activated endeffectors such as electrosurgical electrodes, transducers, sensors,and/or the like. In various embodiments, medical instrument 226 is abiopsy instrument, which may be used to remove sample tissue or asampling of cells from a target anatomic location. Medical instrument226 may be used with an image capture probe also within flexible body216. In various embodiments, medical instrument 226 may be an imagecapture probe that includes a distal portion with a stereoscopic ormonoscopic camera at or near distal end 218 of flexible body 216 forcapturing images (including video images) that are processed by avisualization system 231 for display and/or provided to tracking system230 to support tracking of distal end 218 and/or one or more of thesegments 224. The image capture probe may include a cable coupled to thecamera for transmitting the captured image data. In some examples, theimage capture instrument may be a fiber-optic bundle, such as afiberscope, that couples to visualization system 231. The image captureinstrument may be single or multi-spectral, for example capturing imagedata in one or more of the visible, infrared, and/or ultravioletspectrums. Alternatively, medical instrument 226 may itself be the imagecapture probe. Medical instrument 226 may be advanced from the openingof channel 221 to perform the procedure and then retracted back into thechannel when the procedure is complete. Medical instrument 226 may beremoved from proximal end 217 of flexible body 216 or from anotheroptional instrument port (not shown) along flexible body 216.

Medical instrument 226 may additionally house cables, linkages, or otheractuation controls (not shown) that extend between its proximal anddistal ends to controllably the bend distal end of medical instrument226. Steerable instruments are described in detail in U.S. Pat. No.7,316,681 (filed on Oct. 4, 2005) (disclosing “Articulated SurgicalInstrument for Performing Minimally Invasive Surgery with EnhancedDexterity and Sensitivity”) and U.S. patent application Ser. No.12/286,644 (filed Sep. 30, 2008) (disclosing “Passive Preload andCapstan Drive for Surgical Instruments”), which are incorporated byreference herein in their entireties.

Flexible body 216 may also house cables, linkages, or other steeringcontrols (not shown) that extend between drive unit 204 and distal end218 to controllably bend distal end 218 as shown, for example, by brokendashed line depictions 219 of distal end 218. In some examples, at leastfour cables are used to provide independent “up-down” steering tocontrol a pitch of distal end 218 and “left-right” steering to control ayaw of distal end 281. Steerable catheters are described in detail inU.S. patent application Ser. No. 13/274,208 (filed Oct. 14, 2011)(disclosing “Catheter with Removable Vision Probe”), which isincorporated by reference herein in its entirety. In embodiments inwhich medical instrument system 200 is actuated by a teleoperationalassembly, drive unit 204 may include drive inputs that removably coupleto and receive power from drive elements, such as actuators, of theteleoperational assembly. In some embodiments, medical instrument system200 may include gripping features, manual actuators, or other componentsfor manually controlling the motion of medical instrument system 200.Elongate device 202 may be steerable or, alternatively, the system maybe non-steerable with no integrated mechanism for operator control ofthe bending of distal end 218. In some examples, one or more lumens,through which medical instruments can be deployed and used at a targetsurgical location, are defined in the walls of flexible body 216.

In some embodiments, medical instrument system 200 may include aflexible bronchial instrument, such as a bronchoscope or bronchialcatheter, for use in examination, diagnosis, biopsy, or treatment of alung. Medical instrument system 200 is also suited for navigation andtreatment of other tissues, via natural or surgically created connectedpassageways, in any of a variety of anatomic systems, including thecolon, the intestines, the kidneys and kidney calices, the brain, theheart, the circulatory system including vasculature, and/or the like.

The information from tracking system 230 may be sent to a navigationsystem 232 where it is combined with information from visualizationsystem 231 and/or the preoperatively obtained representations (e.g.,models) to provide the physician, clinician, or surgeon or otheroperator with real-time position information. In some examples, thereal-time position information may be displayed on display system 110 ofFIG. 1 for use in the control of medical instrument system 200. In someexamples, control system 116 of FIG. 1 may utilize the positioninformation as feedback for positioning medical instrument system 200.Various systems for using fiber optic sensors to register and display asurgical instrument with surgical images are provided in U.S. patentapplication Ser. No. 13/107,562, filed May 13, 2011, disclosing,“Medical System Providing Dynamic Registration of a Model of an AnatomicStructure for Image-Guided Surgery,” which is incorporated by referenceherein in its entirety.

In some examples, medical instrument system 200 may be teleoperatedwithin medical system 100 of FIG. 1. In some embodiments,teleoperational manipulator assembly 102 of FIG. 1 may be replaced bydirect operator control. In some examples, the direct operator controlmay include various handles and operator interfaces for hand-heldoperation of the instrument.

FIGS. 3A and 3B are simplified diagrams of side views of a patientcoordinate space including a medical instrument mounted on an insertionassembly according to some embodiments. As shown in FIGS. 3A and 3B, asurgical environment 300 includes a patient P is positioned on platform302. Patient P may be stationary within the surgical environment in thesense that gross patient movement is limited by sedation, restraint,and/or other means. Cyclic anatomic motion including respiration andcardiac motion of patient P may continue, unless patient is asked tohold his or her breath to temporarily suspend respiratory motion.Accordingly, in some embodiments, data may be gathered at a specific,phase in respiration, and tagged and identified with that phase. In someembodiments, the phase during which data is collected may be inferredfrom physiological information collected from patient P. Within surgicalenvironment 300, a point gathering instrument 304 is coupled to aninstrument carriage 306. In some embodiments, point gathering instrument304 may use EM sensors, shape-sensors, and/or other sensor modalities.Instrument carriage 306 is mounted to an insertion stage 308 fixedwithin surgical environment 300. Alternatively, insertion stage 308 maybe movable but have a known location (e.g., via a tracking sensor orother tracking device) within surgical environment 300.

Instrument carriage 306 may be a component of a teleoperationalmanipulator assembly (e.g., teleoperational manipulator assembly 102)that couples to point gathering instrument 304 to control insertionmotion (i.e., motion along the A axis) and, optionally, motion of adistal end 318 of an elongate device 310 in multiple directionsincluding yaw, pitch, and roll. Instrument carriage 306 or insertionstage 308 may include actuators, such as servomotors, (not shown) thatcontrol motion of instrument carriage 306 along insertion stage 308.

Elongate device 310 is coupled to an instrument body 312. Instrumentbody 312 is coupled and fixed relative to instrument carriage 306. Insome embodiments, an optical fiber shape sensor 314 is fixed at aproximal point 316 on instrument body 312. In some embodiments, proximalpoint 316 of optical fiber shape sensor 314 may be movable along withinstrument body 312 but the location of proximal point 316 may be known(e.g., via a tracking sensor or other tracking device). Shape sensor 314measures a shape from proximal point 316 to another point such as distalend 318 of elongate device 310. Point gathering instrument 304 may besubstantially similar to medical instrument system 200.

A position measuring device 320 provides information about the positionof instrument body 312 as it moves on insertion stage 308 along aninsertion axis A. Position measuring device 320 may include resolvers,encoders, potentiometers, and/or other sensors that determine therotation and/or orientation of the actuators controlling the motion ofinstrument carriage 306 and consequently the motion of instrument body312. In some embodiments, insertion stage 308 is linear. In someembodiments, insertion stage 308 may be curved or have a combination ofcurved and linear sections.

FIG. 3A shows instrument body 312 and instrument carriage 306 in aretracted position along insertion stage 308. In this retractedposition, proximal point 316 is at a position L0 on axis A. In thisposition along insertion stage 308 an A component of the location ofproximal point 316 may be set to a zero and/or another reference valueto provide a base reference to describe the position of instrumentcarriage 306, and thus proximal point 316, on insertion stage 308. Withthis retracted position of instrument body 312 and instrument carriage306, distal end 318 of elongate device 310 may be positioned just insidean entry orifice of patient P. Also in this position, position measuringdevice 320 may be set to a zero and/or the another reference value(e.g., I=0). In FIG. 3B, instrument body 312 and instrument carriage 306have advanced along the linear track of insertion stage 308 and distalend 318 of elongate device 310 has advanced into patient P. In thisadvanced position, the proximal point 316 is at a position L1 on theaxis A. In some examples, encoder and/or other position data from one ormore actuators controlling movement of instrument carriage 306 alonginsertion stage 308 and/or one or more position sensors associated withinstrument carriage 306 and/or insertion stage 308 is used to determinethe position Lx of proximal point 316 relative to position L0. In someexamples, position LX may further be used as an indicator of thedistance or insertion depth to which distal end 318 of elongate device310 is inserted into the passageways of the anatomy of patient P.

In an illustrative application, a medical instrument system, such asmedical instrument system 200, may include a robotic catheter system foruse in lung biopsy procedures. A catheter of the robotic catheter systemprovides a conduit for tools such as endoscopes, endobronchialultrasound (EBUS) probes, and/or biopsy tools to be delivered tolocations within the airways where one or more targets of the lungbiopsy, such as lesions, nodules, tumors, and/or the like, are present.When the catheter is driven through anatomy, typically an endoscope isinstalled such that a clinician, such as surgeon O, can monitor a livecamera feed of a distal end of the catheter. The live camera feed and/orother real-time navigation information may be displayed to the clinicianvia a graphical user interface. An example of a graphical user interfacefor monitoring the biopsy procedure is covered in U.S. ProvisionalPatent Application No. 62/486,879, entitled “Graphical User Interfacefor Monitoring an Image-Guided Procedure and filed Apr. 18, 2017, whichis incorporated by reference above.

Before the biopsy procedure is performed using the robotic cathetersystem, pre-operative planning steps may be performed to plan the biopsyprocedure. Pre-operative planning steps may include segmentation ofimage data, such as a patient CT scan, to create a 3D model of anatomy,selecting targets within the 3D model, determining airways in the model,growing the airways to form a connected tree of airways, and planningtrajectories between the targets and the connected tree. One or more ofthese steps may be performed on the same robotic catheter system used toperform the biopsy. Alternately or additionally, planning may beperformed on a different system, such as a workstation dedicated topre-operative planning. The plan for the biopsy procedure may be saved(e.g., as one or more digital files) and transferred to the roboticcatheter system used to perform the biopsy procedure. The saved plan mayinclude the 3D model, identification of airways, target locations,trajectories to target locations, routes through the 3D model, and/orthe like.

Illustrative embodiments of a graphical user interface for planning amedical procedure, including but not limited to the lung biopsyprocedure described above, are provided below. The graphical userinterface may include a plurality of modes including a data selectionmode, a hybrid segmentation and planning mode, a preview mode, a savemode, a management mode, and a review mode. Some aspects of thegraphical user interface are similar to features described in U.S.Provisional Patent Application No. 62/357,217, entitled “Graphical UserInterface for Displaying Guidance Information During and Image-GuidedProcedure” and filed Jun. 30, 2016, and U.S. Provisional PatentApplication No. 62/357,258, entitled “Graphical User Interface forDisplaying Guidance Information in a Plurality of Modes During andImage-Guided Procedure” and filed Jun. 30, 2016, which are herebyincorporated by reference in their entirety.

FIGS. 4-9 are simplified diagrams of a graphical user interface 400displayable on a display system, such as display system 110 and/or adisplay system of an independent planning workstation, according to someembodiments. Graphical user interface 400 displays informationassociated with planning a medical procedure in one or more views thatare viewable to a user, such as surgeon O. Although illustrativearrangements of views is depicted in FIG. 4-9, it is to be understoodthat graphical user interface 400 may display any suitable number ofviews, in any suitable arrangement, and/or on any suitable number ofscreens. In some examples, the number of concurrently displayed viewsmay be varied by opening and closing views, minimizing and maximizingviews, moving views between a foreground and background of graphicaluser interface 400, switching between screens, and/or otherwise fully orpartially obscuring views. Similarly, the arrangement of theviews—including their size, shape, orientation, ordering (in a case ofoverlapping views), and/or the like—may vary and/or may beuser-configurable.

In some examples, graphical user interface 400 may include one or moreheaders, footers, sidebars, menus, message bars, pop-up windows, and/orthe like. As depicted in FIGS. 4-9, graphical user interface 400includes a dynamic header 410 that is updated based on the mode ofgraphical user interface 400. In various examples, header 400 mayinclude a drop-down control menu, a page title, navigation controls(e.g., a proceed button and/or a back button), patient information, asearch bar, and/or the like.

FIG. 4 illustrates graphical user interface 400 in a data selection modeaccording to some embodiments. The data selection mode is used to selecta data source, a patient, and/or image data to use when planning themedical procedure. Accordingly, graphical user interface 400 in thesource selection mode may include a data source selector 420, a patientselector 430, and a data selector 440. As depicted in FIG. 4, datasource selector 420 includes options to load data from a USB device, aDVD, and/or a network. It is to be understood that data may be loadedfrom a variety of other sources, including external and/or local sources(e.g., a local hard drive). Patient selector 430 includes a list ofpatients whose image data is available from the selected data source.Various patient attributes may be displayed in the list, include thepatient name, gender, date of birth, unique patient ID, and/or the like.Data selector 440 includes a list of image data available for theselected patient from the selected data source. Various attributes ofthe data may be displayed in the list, including the data description,the date the data was acquired, and/or a suitability rating indicatingthe suitability of the image data for planning the medical procedure.The suitability rating may be qualitative and/or quantitative and may beassigned manually and/or automatically. The rating may be presented as anumeric score, a star rating, a percentile, a symbolic representation,and/or the like. In some examples, the suitability rating may bedetermined based on the quality of imaging technology used to acquirethe image data. Once the image data is selected, the user may proceed toplan the medical procedure using the selected image data. For example,the user may click and/or tap a load button of a navigation panel 450 toproceed.

FIGS. 5A-5G illustrate graphical user interface 400 in a hybridsegmentation and planning mode according to some embodiments.Segmentation is a process that analyzes image data, such as the imagedata selected in the data selection mode, and creates a 3D model fromthe data. Examples of automated techniques for performing segmentationof CT data are described in U.S. patent application Ser. No. 14/845,031,entitled “Systems and Methods for Pre-Operative Modeling,” which ishereby incorporated by reference in its entirety. The segmentationprocess generally occurs over a period of time, e.g., one to threeminutes, which may vary depending on a number of factors including thequality of the CT image data, the size and/or complexity of the CT imagedata, the level of detail in the 3D model, the available computingresources, and/or the like. In some examples, the hybrid segmentationand planning mode of graphical user interface 400 may allow the user toplan a medical procedure based on the image data and/or the 3D modelwhile the segmentation process is occurring and before the 3D model iscomplete. Accordingly, the process of planning the medical procedure maybe accelerated because the user is able to begin planning the medicalprocedure without waiting for the potentially lengthy segmentationprocess to finish.

In some embodiments, graphical user interface 400 in the hybridsegmentation and planning mode may be split into one or more frames. Asillustrated in FIGS. 5A-5G, graphical user interface 400 includes acontrol frame 510 and a canvas frame 520. Control frame 510 provides aset of controls and/or indicators for planning the medical procedure. Insome examples, control frame 510 may provide controls for adding,viewing, modifying, and/or deleting one or more features of the modeland/or the plan, such as targets, paths, airways, trajectories, and/orhazards. In some examples, control frame 510 may provide controls toundo and/or redo recent changes to the plan. In some examples, controlframe 510 may provide a segmentation progress indicator 515 based on howfar along the segmentation process is. Segmentation progress indicator515 may be formatted as a progress bar, an elapsed time indicator, anestimated remaining time indicator, and/or any other suitable indicatorof segmentation progress. In some embodiments, segmentation progressindicator 515 may disappear when segmentation is complete.

In some embodiments, graphical user interface 400 in the hybridsegmentation and planning mode may include a canvas frame 520. Asillustrated in FIGS. 5A-5G, canvas frame 520 provides a workspace 522for selecting, viewing, and/or interacting with image data and/or modeldata. Illustrative functions that may be performed via workspace 522include adding, modifying, and/or deleting features (e.g., targets,paths, and/or hazards) of the plan, manipulating the 3D model, and/orverifying the accuracy of the segmentation process. To accommodate thesefunctions, workspace 522 may transition among a plurality of interactiveviews, including a selection view, one or more image views, and one ormore model views.

In some examples, canvas frame 520 may include a tool selector 524 thatprovides a list of available tools. As depicted in FIGS. 5A-5G, the listof tools includes a move tool, a magnifier tool, a window/level tool, anobject drawing tool, a line drawing tool, a trimming tool, a hazardtool, an angle and/or distance measurement tool, an undo/redo tool,and/or the like. In some examples, certain tools may be enabled and/ordisabled based on the current view displayed in workspace 522. Forexample, a tool that is not used by the current view may be hidden,grayed out, and/or otherwise not selectable. In some examples, clickingon a tool may cause a menu to appear with a list of sub-tools. Forexample, the object drawing tool may include sub-tools for drawingvarious 2D and/or 3D objects, such as freeform objects, predefined 2Dshapes (e.g., circles, rectangles, ellipses, etc.), 3D shapes (e.g.,spheres, 3D ellipsoids, etc.), and/or the like. In some examples, toolselector 524 may include tools for semi-automatically detecting objectsin the underlying image data (e.g., clicking a point in the image dataand using edge detection techniques to automatically identify acorresponding object). Although tool selector 524 is depicted as asidebar, it is to be understood that tool selector 524 may be positionedand/or displayed in a variety of formats, including a palette, header,footer, dropdown menu, auto-hiding menu, and/or the like. In someembodiments, tool selector 524 may be omitted, such as when toolselection is performed using keyboard shortcuts.

FIG. 5A illustrates an example of a selection view displayed inworkspace 522. In the selection view, a set of selections 531-536 arepresented as a thumbnail grid. Each of selections 531-536 corresponds toa different rendering of image data and/or representation (e.g., model)data. The renderings may vary based on their perspective, zoom level,data type, styles, and/or the like. For example, renderings of imagedata may be provided from a variety of perspectives including atransverse, sagittal, coronal, and/or virtual endoscopic perspective. Inthe example depicted in FIG. 5A, selections 535 and 536, whichcorrespond to rendered representation data, display waiting indicatorsbecause the segmentation is not yet complete and the representation isnot ready to display. On the other hand, selections 531-534, whichcorrespond to renderings of image data, are populated with actual imagedata because graphical user interface 400 allows can display and/orreceive interactive inputs for the image data before segmentation iscomplete. Once a selection for a rendering is received, the selectedrendering can be displayed, interactive inputs received or both via aninteractive window. For example, a user input, such as a click or a tapcan be receive via an expand view button 537 of the selected renderingto proceed. In some examples, expand view button 537 may appear whendetecting the user or an object held by the user hovering over thecorresponding selection and may disappear otherwise. Although selections531-536 are depicted as being arranged in a thumbnail grid, a variety ofalternatives are possible, such as a list of selections.

FIGS. 5B-5F illustrate examples of an interactive window 541 displayedin workspace 522. Interactive window 541 displays the rendering that wasselected using the selection view. In some examples, a selection sidebar542 may be displayed alongside interactive window 541 to allow the userto change to a different rendering without returning to the selectionview. For example, selection sidebar 542 may display a scrollable columnof thumbnail images generally corresponding to selections 531-536, withthe current selection being identified by a blue border.

As depicted in FIG. 5B, raw image data 543 (e.g., CT image data) isdisplayed a first color palette, such as grayscale, and segmentationdata 544 (e.g., detected airways) is displayed in a contrasting color orshade, such as pink. In some examples, when segmentation is still inprogress, segmentation data 544 may be dynamically updated to reflectsegmentation progress. For example, new pink regions may dynamicallyappear in interactive window 541 as new airways are detected over time.

In some examples, interactive window 541 may display one or morefeatures of the plan of the medical procedure, such as targets, paths,and/or hazards. The features may include user input-based features,automatically extracted features, semi-automatically extracted featuresand/or the like. According to some embodiments, changes to the one ormore features made in a particular rendering may be dynamicallypropagated to other renderings. For example, a target added in onerendering may automatically appear in the other renderings, includingthumbnail images of selection sidebar 542.

As depicted in FIG. 5B, the features include a target 550 identifiedusing a circle tool over a lesion visible in the underlying image data.The size, shape, and/or position target 550 may be adjusted to capturethe shape of the lesion with the desired level of accuracy. When thesize, shape, and/or position of target 550 is adjusted, statistics 552corresponding to target 550 of are updated in control frame 510. In someexamples, target 550 may be named, renamed, and/or deleted usingcontrols provided in control frame 510. In some examples, controlsprovided in control frame 510 enable identification of additionaltargets after the first target has been identified by detectingcorresponding user inputs via the control frame 510. Additionally oralternately, tool selector 524 may include one or more tools for adding,modifying, and/or deleting targets. In some examples, identifying target550 in one rendering may automatically cause an updated representationof target 550 to appear in other renderings. Consequently, theparameters of target 550 can be adjusted from multiple perspectivesbased on detecting user inputs from the available renderings that areswitchably displayed.

In FIG. 5C, interactive window 541 includes a trajectory 560 betweentarget 550 and an exit location 562. Exit location 562 corresponds to apoint where a medical instrument exits the anatomical passagewaysdetected by the segmentation process to reach target 550. In someexamples, exit location 562 is a closest point from a closest anatomicalpassageway to target 550. Trajectory 560 represents the trajectory for amedical instrument positioned at exit location 562 to perform one ormore interventional steps at target 550. The instrument may, forexample, puncture through the lumen of the anatomical passageway at theexit location. For example, the medical instrument may include a biopsyneedle, an ablation tool, a chemical delivery system, an ultrasoundprobe, and/or the like. In some examples, the medical instrument mayhave a maximum trajectory length. For example, a biopsy needle may notbe able to perform a biopsy at a target that is more than 3 cm from exitlocation 562. Consequently, when the length of trajectory 560 is greaterthan 3 cm, graphical user interface 400 may display an out of rangewarning 564. In some embodiments, out of range warning 564 can beprovided based on a threshold, which may include a fixed threshold valueand/or a variable threshold that is set based on, e.g., the type of toolwhich is to be used to access the target. For example, a biopsy needlemay provide a different insertion depth than an ablation tool or animaging probe, in which case the threshold may vary accordingly. Inanother example, different types of biopsy needles could providedifferent insertion depths. The user could input the type of tool beingused or the tool could be automatically detected by the system. Asdepicted in FIG. 5C, out of range warning 564 is presented as a messagein control frame 510.

In some embodiments, multiple trajectories to a given target may beidentified, such as an alternate trajectory to be used when trajectory560 is found to be unreachable and/or otherwise inadequate for useduring the medical procedure. Consistent with such embodiments, controlframe 510 may include controls for adding an alternate trajectory totarget 550. Additionally or alternately, tool selector 524 may includeone or more tools for adding, modifying, and/or deleting trajectories.

In FIG. 5D-5F, interactive window 541 includes a hazard fence 570.Hazard fence 570 is used to facilitate trajectory planning byidentifying vulnerable portions of the anatomy that are in the vicinityof the target location. Examples of vulnerable portions of the anatomymay include blood vessels, lung pleura, large bullae, and/or the like.For example, puncturing the lung pleura during the medical procedurecould cause dangerous pneumothorax to the patient. Consistent with suchembodiments, the exit location and/or the trajectory between the exitlocation and the target location may be constrained to avoid thevulnerable portion of the anatomy. For example, a trajectory may beinvalid when it passes within a threshold distance of a vulnerableportion of the anatomy, breaches a vulnerable portion of the anatomy,and/or the like. In the examples depicted in FIGS. 5D-5F, hazard fence570 provides a warning to protect a portion of the lung pleura that isclose to the target location from being punctured when using the plannedtrajectory. As depicted in FIGS. 5D-5F, hazard fence 570 is placed usingthe hazard tool of tool selector 524. Additionally or alternately,hazard fences may be added, modified, and/or removed using controlspresented in control frame 510.

Each of FIGS. 5D-5F illustrates a different style of hazard fence 570.In FIG. 5D, hazard fence 570 is displayed as a planar hazard fence witha pair of control points 571 and 572 used to define a circular disk inthree dimensions. To convey the 3-dimensional aspects of the circulardisk, portions of the circular disk that project out of interactivewindow 541 may be rendered in solid color, whereas portions of thecircular disc that project into interactive window 541 may be renderedin a faded and/or translucent color. In FIG. 5E, hazard fence 570 isdisplayed a conic hazard fence with a pair of outer control points 573and 574 used to define a 3-dimensional circular disk as the base of thecone and a vertex control point 575 used to define the height of a cone.In FIG. 5F, hazard fence 570 is displayed as a hemispherical hazardfence with a triad of control points 576-578 used to define ahemisphere. In FIGS. 5E and 5F, interactive window 541 further includestarget 550 and trajectory 560. When trajectory 560 connects to an exitlocation that is not in the plane of the underlying image (that is, whentrajectory 560 projects into and/or out of interactive window 541) aprojection 579 is displayed to link trajectory 560 to modeledpassageways.

Various other types of hazards may be identified and marked using asuitable indicator, such as hazard fence 570. For example, an anatomicalpassageway may create a tight bend that cannot be traversed by certainmedical instruments, such as a biopsy needle and/or a catheter.Accordingly, the bend may be indicated using a blocking sign such thatthe user knows to plan a different route to the target that avoids thebend. Automatic, manual, and/or semi-automatic techniques may be used todetermine whether a planned route includes any bends that are too tight.For example, given the known physical characteristics of various medicalinstruments to be used in the medical procedure, a bend radius that istoo tight may be automatically identified. Additionally or alternately,a user may visually identify that a bend appears to be too tight, and/ormay perform measurements to confirm that a bend is too tight. In someexamples, candidate routes may be automatically ranked based onuser-defined rules and/or feasibility characteristics of the routes,such as the length of the routes, the tightest bend encountered in theroute, the width of the passageways along the route, the length of thetrajectory between the end of the route and the target, and/or the like.Accordingly, the user may select among the candidate routes based on therankings.

In FIG. 5G, interactive window 541 displays a representation (e.g., amodel) 580 corresponding to the 3D model generated by segmentation ofthe image data. In some examples, representation 580 may not beavailable until segmentation is complete. In some examples, whensegmentation is incomplete, a partial version of model 580 may bedisplayed and may be updated in real-time to reflect ongoingsegmentation progress. In some embodiments, a boundary 581 may bedisplayed around model 580 as a translucent pattern, wire frame pattern,and/or the like. As depicted in FIG. 5G, boundary 581 corresponds to thelung pleura. The appearance of boundary 581 (e.g., color, size, texture,and/or the like) may vary to identify various features. For example,boundary 581 may be colored red to indicate the position of hazard fence570. A variety of features, including target 550, trajectory 560, exitlocation 562, and/or the like, may also be depicted in interactivewindow 541. According to some embodiments, interactive window 541 mayinclude an orientation icon 582 to identify the viewing perspective ofmodel 580 relative to the patient body. In some examples, the appearanceof passageways in model 580 that are along the route to target 550 maybe altered to indicate that they are along the route. For examplepassageways that are on the route may be colored blue, whereas otherpassageways in model 580 may be colored gray.

In some examples, an exit angle selector 583 may be provided in controlframe 510. Exit angle selector 583 provides an adjustment control, suchas a slider, to adjust the position of exit location 562 along ananatomical passageway. Adjusting the position of exit location 562causes a corresponding adjustment to the exit angle of trajectory 560relative to the anatomical passageway. In some examples, it may bedesirable to set the exit angle of trajectory 560 based on a variety offactors and/or metrics, such as a default or ‘rule of thumb’ exit angle(e.g., 45 degrees), the distance between exit location 562 and target550, and/or the distance between exit location 562 and hazard fence 570.Accordingly, exit angle selector 583 may accelerate the process ofdefining trajectory 560 by allowing the user to rapidly test a range ofexit angles and confirm that relevant metrics fall within acceptableranges. For example, exit angle selector 583 may display the value ofthe exit angle (e.g., 37 degrees in the example provided), the distancefrom exit location 562 to target 550 (e.g., 2.4 cm in the exampleprovided), and/or the distance from exit location 562 to hazard fence570 (e.g., 7.4 cm in the example provided). The appearance of exit angleselector 583 (e.g., color, texture, size, font, etc.) may vary to alertthe user when one or more of the relevant metrics are not within apredetermined range and/or do not meet a predetermined threshold. Insome examples, one or more values of the angle adjustment slider may bedisabled when the values are determined to be outside of an acceptablerange.

FIG. 6 illustrates graphical user interface 400 in a preview modeaccording to some embodiments. The preview mode is used to preview theplan of the medical procedure that was prepared in the hybridsegmentation and planning mode. Graphical user interface 400 in apreview mode displays a simulated live endoscopic view 610, a matchingvirtual endoscopic view 620, a global anatomical representation (e.g.,model) view 630, and a reduced anatomical representation (e.g., model)view 640. Simulated live endoscopic view 610 and virtual endoscopic view620 depict renderings of the representation (e.g., model) from insidethe anatomical passages. The renderings are from the perspective of avirtual endoscope that is following the route of the planned medicalprocedure. Simulated live endoscopic view 610 and virtual endoscopicview 620 are generally similar, except simulated live endoscopic view610 includes photorealistic details (e.g., blood vessels in theanatomical lumens) to simulate an actual camera feed from an endoscope,whereas virtual endoscopic view 620 is augmented with directional cuestowards the target location, such as contour lines 621, a route line,arrows, and/or the like. Where the anatomical passageways branch, thepassageways may light up in virtual endoscopic view 620 to indicate thedirection that the user should steer. One or more of simulated liveendoscopic view 610 and virtual endoscopic view 620 may display varioustrajectory metrics, such as the remaining distances to the targetlocation and/or hazards.

Global anatomical model view 630 generally corresponds to the 3Dperspective view of model 580 described in FIG. 5G. As depicted in FIG.6, global anatomical model view 630 includes a model rendering 631 thatincludes the anatomical model, a boundary, multiple target locations, ahazard fence, and an orientation icon. Anatomical model view 630 furtherincludes a depiction of a catheter 632 as a green line. Endoscope views610 and 620 provide matching views from a distal end of catheter 632.

In some embodiments, graphical user interface 400 in the preview modemay display a reduced anatomical model view 640. Reduced anatomicalmodel view 640 provides a simplified overview of the planned route ofthe medical procedure that includes key anatomical features of theroute. A route path 641 is represented as a straight line. A depictionof catheter 632 is overlaid on route path 641 to indicate the progressof catheter 632 along route path 641. An anatomical passageway 643 isrendered as a 2D tiered projection to provide a simplified indication ofthe width of passageway 643. Branches 644 are rendered to show thelocations where they connect to passageway 643, but other details ofbranches 644, such as their various sub-branches, are not rendered. Atarget icon 645 that indicates the exit angle and/or nearby hazards islocated at the distal end of route path 641. When the plan of themedical procedure includes multiple targets and/or paths, a selector 646is included to switch among the multiple targets and/or paths.Embodiments of reduced anatomical representation (e.g., model) views arefurther discussed in U.S. Provisional Patent Application No. 62/486,879,which is incorporated by reference above.

As depicted in FIG. 6, reduced anatomical model view 640 serves as acontroller to allow the user to navigate through a preview of the route.In particular, the distal end of catheter 632 includes a control point650 that can be dragged back and forth along route path 641. As controlpoint 650 is dragged back and forth, endoscope views 610 and 620 areupdated to reflect the viewpoint of catheter 632 at the updated positionof control point 650. Furthermore, the depiction of catheter 632 inglobal anatomical model view 630 is updated to reflect the shape ofcatheter 632 at the updated position of control point 650. In theexample depicted in FIG. 6, control point 650 is displayed as atriangular cone representing the projected view of the endoscope fromthe distal end of the catheter. In alternative embodiments, controlpoint 650 could be various shapes and sizes.

Once the one or more routes of the plan have been previewed, theclinician may proceed to save the plan. For example, the clinician mayclick and/or tap a next step button of header 410 to proceed.Alternately, the clinician may revert to an earlier stage of theplanning process to make alterations as desired.

FIG. 7 illustrates graphical user interface 400 in a save mode accordingto some embodiments. The save mode is used when the planned medicalprocedure is complete and/or ready to transfer to a medical instrumentto perform the medical procedure. A set of options 710 are presented viagraphical user interface 400. Options 710 may include a transfer option,a discard option, a delete option, and/or a save option. The save optionmay include saving the plan locally, saving to an external device,and/or transmitting the plan over the network, e.g., to a cloud storagefacility. One or more options may require an external storage device tobe installed. For example, the transfer option may require a storagedevice (e.g., a USB device) that is compatible with the medicalinstrument that the plan is to be transferred to. Accordingly, a message720 may be displayed to inform the user of the applicable storage devicerequirements.

FIG. 8 illustrates graphical user interface 400 in a management modeaccording to some embodiments. The management mode is used to manageavailable plans. The available plans may be stored locally, on anexternal drive, and/or may be available for download over a network. Aselection grid 810 is displayed that includes a thumbnail representationof the plan (e.g., a rendering of the representation (e.g, model),planned route, target locations, and/or the like). Additionally oralternately, selection grid 810 may include patient data such as thepatient name, date of birth, and/or the like. In some examples, aprocedure being planned can be reloaded and viewed using the previewmode of graphical user interface 400. In some examples, a procedurebeing planned can be saved at any time and reloaded to continue work ata later time using the hybrid segmentation and planning mode ofgraphical user interface 400. Accordingly, selection grid 810 mayinclude a status of each plan (e.g., transferred, planned, started,and/or the like) and/or an indicator of when the plan was last saved. Insome examples, a selected plan may be reviewed, deleted, and/ortransferred. In some embodiments, selection grid 810 may include anoption to create a new plan. When the new plan is selected, graphicaluser interface 400 may proceed to the data selection mode describedpreviously in FIG. 4.

FIG. 9 illustrates graphical user interface 400 in a review modeaccording to some embodiments. The review mode is used to review arecord of a completed medical procedure. After a medical procedure isperformed using a given plan, the record of the procedure may be savedand transferred to the planning workstation. In some embodiments, therecorded procedure file may include video of a live endoscopic imagecaptured during the procedure, a correlated virtual endoscopic image, ananatomical representation (e.g, model) showing catheter movements duringthe procedure, notes taken by the clinician during the procedure, and/orthe like. Accordingly, a viewer 910 may be displayed that includesplayback controls (e.g., play, pause, zoom, and/or the like), snapshotcontrols, annotation and/or bookmark controls, and/or the like.

FIG. 10 is a simplified diagram of a method 1000 for planning a medicalprocedure according to some embodiments. According to some embodimentsconsistent with FIGS. 1-9, method 1000 may be used to operate agraphical user interface, such as graphical user interface 400, in aplurality of modes including a data selection mode, a hybridsegmentation and planning mode, a preview mode, a save mode, amanagement mode, and review mode. In some embodiments, the graphicaluser interface is interactive and may receive user inputs via a mouse,keyboard, touch, stylus, trackball, joystick, speech commands, virtualreality interface, and/or the like.

At a process 1010, data is selected via the graphical user interface inthe data selection mode. According to some embodiments, selecting thedata includes selecting a data source using a data source selector, suchas data source selector 420, selecting a patient using a patientselector, such as patient selector 430, and selecting the data using adata selector, such as data selector 440. The selection may be confirmedby engaging a load button on the graphical user interface. Data caninclude imaging data, such as CT data and/or any other type of imagingor patient data.

At a process 1020, a medical procedure is planned via the graphical userinterface in the hybrid segmentation and planning mode. According tosome embodiments, the data selected at process 1010 includes image datathat is segmented to generate an anatomical representation (e.g, model)based on extracted passageways. Concurrently during segmentation, themedical procedure is planned by receiving user inputs defining featuresof the plan, such as targets, hazards, and/or paths. In some examples,an interactive window, such as interactive window 541, may provide aninterface for the user to add, modify, and/or delete features from theplan. When segmentation has progressed such that the representation(e.g, model) is ready for viewing, the interactive window may be used toview and/or interact with the representation (e.g, model). In someexamples, a target may not have any extracted passageways close enoughto draw a valid trajectory (e.g., a trajectory that is shorter than themaximum trajectory length) between the target and an extractedpassageway. Accordingly, the user may manually identify and add a nearbypassageway to the representation (e.g, model). An exemplary method formanually adding a connected passageway to the representation (e.g,model) is described in greater detail below with reference to FIG. 11.

At a process 1030, the planned medical procedure is previewed via thegraphical user interface in the preview mode. According to someembodiments, previewing the medical procedure may include viewing a livesimulated endoscope view, such as live simulated endoscope view 610, avirtual endoscope view, such as virtual endoscope view 620, ananatomical model view, such as anatomical model view 630, and/or areduced model view, such as reduced model view 640. According to someembodiments, the reduced model view may include a control point, such ascontrol point 650, to scroll back and forth through the preview of themedical procedure.

At a process 1040, the planned medical procedure is transferred to amedical instrument via the graphical user interface in a save mode.According to some embodiments, transferring the planned medicalprocedure may include installing a storage device that is compatiblewith the planning workstation and the medical instrument. In someexamples, a message may be displayed via the graphical user interface toalert the user that compatibility is required. In some examples, theplanned medical procedure may be saved during process 1040. In someexamples, the planned medical procedure may be transferred to a roboticcatheter system. In some examples, after process 1040, method 1000 mayproceed to a process 1050 to perform the medical procedure in accordancewith the plan.

FIG. 11 is a simplified diagram of a method 1100 for modifying ananatomical model to provide access to a target of a medical procedureaccording to some embodiments. According to some embodiments, method1100 may be performed using a graphical user interface, such asgraphical user interface 400, in a hybrid segmentation and planningmode. In some examples, method 1100 may generally be performed aftersegmentation is complete and the anatomical model is available to beviewed and/or manipulated.

In general, the passageways of interest to a user are those that arecontinuously navigable by an instrument from a main passageway, such asthe trachea, through various branches to an exit point at a passagewaynear the target. In some cases, automatic segmentation may not detectall such passageways. Accordingly, the set of passageways that areconnected to the model generated by segmentation is incomplete. When theinitial model does not provide satisfactory access to the target (e.g.,when the closest exit point is not within a threshold distance, such as3 cm, as previously described with respect to FIG. 5C), the user maydesire to connect one or more passageways that are initially unconnectedfrom the model. In some cases, automatic segmentation may detectpassageways that are not of interest to the user, such as passagewaysthat do not lead to the target. Method 1100 provides examples oftechniques for identifying passageways of interest that were notdetected by automatic segmentation and connecting the unconnectedpassageways to the model. Method 1100 further provides examples oftechniques for trimming passageways from the model that are not ofinterest to the user.

At a process 1110, a distance between the target and the nearestconnected passageway is measured. According to some embodiments, thedistance may be measured automatically, e.g., in response to the userdefining the target via the graphical user interface. In some examples,the distance may be measured via the graphical user interface byclicking on the target and the nearest connected passageway.

At a process 1120, it is determined whether the measured distance isgreater than a predetermined threshold. In some examples, thepredetermined threshold may correspond to the maximum range of a medicaltool used in the medical procedure, such as a biopsy needle. In someexamples, the predetermined threshold may be a fixed value, such as 3cm, and/or may be variable based on factors such as the type and/ormodel of the medical tool being used. At a process 1130, it isdetermined that the measured distance is less than the predeterminedthreshold. The model may be saved and method 1100 may terminate atprocess 1130 because the existing model provides satisfactory access tothe target. When the distance is greater than the predeterminedthreshold, method 1100 may proceed to a process 1140 for identifying anunconnected passageway that is close to the target and growing the modelto include the identified passageway, as described in greater detailbelow with reference to FIG. 12. Once the identified passageway has beenconnected to the model, method 1100 may repeat process 1110 and process1120 until a passageway has been connected to the model that is withinthe predetermined threshold distance from the target.

In some embodiments, the distance between the target and the nearestconnected passageway may not be the only consideration in determining ifthe model provides sufficient airways to reach a target. In someexamples, other factors affecting an exit point from the closest airwayto the target can be considered. Such factors can include satisfactoryexit angle from the exit point, presence of tight radius bends that mustbe navigated through connecting airways to reach the exit point,diameter size of anatomical passageways and/or potential hazards betweenthe exit point and the target. A different path through differentairways may be selected based on these considerations. Thus, in analternative embodiment, optional processes 1150 a, 1150 b, and 1150 ccan be completed to evaluate the other factors with respect to aselected passageway, determine whether the other factors aresatisfactory, and select an alternative passageway when the otherfactors are unsatisfactory.

FIG. 12 is a simplified diagram of a method 1200 for growing ananatomical model to provide access to a target of a medical procedureaccording to some embodiments. At a process 1210, an interactive imageis displayed via a graphical user interface, such as graphical userinterface 400. The interactive image depicts image data, connectedpassageways within the image data, and the target within the image data.At a process 1220, a user input is received to identify at least a partof an unconnected passageway that is closer to the target than thenearest connected passageway. In one example, the anatomical model canbe grown using method 1200 based on detected identification ofunconnected passageways adjacent to or near the connected passageways ofthe model and progressively working towards the target in a “forward”approach. In alternate examples, method 1200 can be used to grow theanatomical model based on detected identification of unconnectedpassageways adjacent to or near the target and progressively workingtowards the connected passageways of the model in a “backward’ approach.

At a process 1230, when a suitable unconnected passageway is notidentified in the initially displayed interactive image, the interactiveimage may be searched by iteratively rotating the interactive image anddetermining whether the unconnected passageway is visible. At a process1240, a rotation point is defined in the graphical user interface byselecting a point in the image data (e.g., by double clicking thepoint). In some examples, the rotation point is displayed by placingcrosshairs on the interactive image. In one example, the rotation pointis chosen as a point along the closest connected airway to the target.In another example the rotation point is chosen as a point on thetarget. In further examples, the rotation point can be any point in theinteractive image. At a process 1250, the interactive image is rotatedabout the rotation point. In some examples, the rotation point provides360 degree rotation in three dimensions about the rotation point. At aprocess 1260, it is determined whether an unconnected passageway isidentified in the interactive image. When an unconnected passageway isnot identified, a new rotation point is selected at a process 1270 andthe interactive image is rotated about the new rotation point in orderto identify an unconnected passageway. Processes 1240-1270 may berepeated until an unconnected passageway is identified.

When an unconnected passageway is identified in the interactive image atprocess 1230 or process 1260, a user input may be received thatidentifies the unconnected passageway (e.g., a click and/or a tap) at aprocess 1280 and the unconnected passageway is connected to the model.In some embodiments, the unconnected passageway may be connected to themodel automatically, using segmentation software to trace the passagewayto a connection point with the model. When process 1280 is complete,method 1200 may return to method 1100 to determine whether the newlyconnected passageway provides satisfactory access to the target.Processes 1110-1150 may be repeated until satisfactory access to thetarget is achieved

According to some embodiments, an unconnected passageway is partiallyidentified when rotating the interactive image about the rotation pointduring process 1250. After rotating the interactive image to a statewhere the unconnected passageway is partially identified, it can behelpful to limit the rotation to rotation about an axis of rotation,rather than unconstrained 3D rotation about the rotation point. The axisof rotation is defined in the graphical user interface by drawing a linebetween the rotation point and a second point, such as the targetlocation. Limiting the rotation to an axis of rotation may enhanceusability relative to rotation about the rotation point alone.Consistent with such embodiments, searching for the unconnectedpassageway may proceed by iteratively repeating processes 1240-1270using any combination of rotation points (when unconstrained rotation isdesired) and axes of rotation (when limited rotation is desired).

Beginning with a point of rotation can provide some advantages tobeginning with an axis of rotation. For example, if the user was toinitially provide an axis of rotation and a given unconnected passagewayis positioned in an orientation that is orthogonal to the axis ofrotation, the user would not see any variation in the appearance of theunconnected passageway when rotating the interactive image, i.e. theunconnected passageway would appear as a circle and when rotating aboutthe selected axis of rotation, the unconnected passageway would alwaysappear as a circle. Using a single point of rotation would provide 360degrees of rotation in three dimensions. Thus, regardless of the initialorientation of the unconnected passageway, the user would eventuallyrotate the interactive image in a manner where the unconnectedpassageway would be visible. Additionally, selecting a new axis ofrotation can prove difficult. The user would have no guidance in how todraw a new line. However, changing a point of rotation is simple in thatit is only selecting a single point. The rotation point could beselected at a position close to a connected passageway but, if thatproves to be an insufficient point of rotation, the rotation point couldbe easily moved to be at the target.

In another example, referring again to FIG. 11 at process 1140, themodel can alternatively be grown when the user draws a line from theclosest connected passageway to the target. Using automated techniques,structures that appear to correspond to unconnected passageways may bedetected, and the closest such structure may be connected to the model.The user can iteratively continue drawing a line from the most recentlyconnected passageway to the target while additional unconnectedpassageways are detected and connected to the model, until the closestconnected passageway falls within the threshold established in process1120. In one example, when a passageway that is unacceptable to the useris connected to the model, the user may select the airway and erase ordelete it from the model

At any time during segmentation and/or during growth of the model, themodel may be trimmed. In some examples, one or more connectedpassageways may be determined to be extraneous. For example, a connectedpassageway may be determined to lead far from the target and/orotherwise serve little and/or no purpose relevant to the medicalprocedure. In some examples, it may be desirable to disconnect anddelete extraneous passageways from the model. For example, disconnectingthe extraneous passageways may reduce visual clutter and/or may improveperformance (e.g., improve load times, rendering times, and/or the like)by reducing the size of the model. Consequently, a trimming tool may beprovided to the user via graphical user interface. For example, thetrimming tool may be selected via a tool selector, such as tool selector524. When the trimming tool is enabled and a user input is received thatidentifies an extraneous passageway (e.g., a user click and/or tap) thepassageway may be disconnected and deleted from the model. In someexamples, the passageway may be disconnected at the point identified bythe user. In some examples, the passageway may be disconnected at thenearest point of connection to the model. In some examples, theidentified passageway may be disconnected along with any sub-branches ofthe identified passageway. According to some embodiments, trimming maybe performed at any time during methods 1100 and/or 1200, and/or as astandalone process independent from methods 1100 and/or 1200.

FIG. 13 is a simplified diagram of a method 1300 for planning a medicalprocedure using a graphical user interface according to someembodiments. According to some embodiments consistent with FIGS. 4-9,method 1300 may be performed using graphical user interface 400 in thehybrid segmentation and planning mode, as described in FIGS. 5A-5G. Insome examples, method 1300 and/or various processes thereof may beperformed before, during, and/or after segmentation of image data togenerate a model.

At a process 1210, an interactive window, such as interactive window541, is provided for a user to create a plan for a medical procedure.The interactive window may be displayed via a display system and mayinteractivity via a user interface such as a mouse, trackball, joystick,touch screen, natural user interface (e.g., voice, gestures),augmented/virtual reality interface, and/or the like. According to someembodiments, the interactive window may be displayed in conjunction withone or more other views, such as a tool selector (e.g., tool selector524), a selection sidebar (e.g. selection sidebar 542), a control frame(e.g. control frame 510), and/or the like.

At a process 1310, image data is displayed via the graphical userinterface. In some examples, the image data may correspond to raw imagedata of a patient (e.g., CT data). The image data may be previouslyselected in a data selection mode of the graphical user interface. Insome examples, the image data may be displayed concurrently while theimage data is being segmented using a background segmentation process.Segmentation data generated by the segmentation process (e.g., airwaysdetected in the image data) may be overlaid on the image data. Forexample, the image data may be displayed in a first color palette, suchas greyscale, and the segmentation data may be displayed in acontrasting color, such as pink. As segmentation of the image dataproceeds, the displayed segmentation data may be updated to reflect thesegmentation progress.

At a process 1320, a first user input is received that defines one ormore features of the plan within the displayed image data. According tosome embodiments, the one or more features of the plan may include atarget of the medical procedure, a hazard of the medical procedure,and/or the like. In some examples, the target may be defined using anobject placement tool with a suitable shape (e.g., a circle tool and/ora 3D ellipse tool) provided by the graphical user interface. In someexamples, the hazard may be defined using a hazard fence placement toolwith a suitable shape (e.g., a 3D circular disk, a conic hazard fence,and/or a hemispherical hazard fence) and/or suitable control points fordefining the hazard fence. Examples of hazards may include vulnerableportions of the anatomy (e.g., lung pleura, blood vessels, large bullae,and/or the heart), and/or excessive bend in an anatomical passageway(e.g., a bend that is too tight to accommodate passage of a medicalinstrument, such as a biopsy needle).

At a process 1330, an interactive image is displayed via the graphicaluser interface. The interactive image includes the image data, connectedanatomical passageways detected by segmentation of the image data, andthe one or more features defined during process 1320. The connectedanatomical passageways form a tree in which each branch is reachablefrom a main passageway, such as a trachea. Accordingly, the connectedanatomical passageways are accessible to a medical instrument insertedvia the main passageway. A user may interact with the interactive imagevia a user interface such as a mouse, trackball, joystick, touch screen,natural user interface (e.g., voice, gestures), augmented/virtualreality interface, and/or the like. According to some embodiments, theinteractive image may be displayed in conjunction with one or more otherviews, such as a tool selector (e.g., tool selector 524), a selectionsidebar (e.g. selection sidebar 542), a control frame (e.g. controlframe 510), and/or the like.

At a process 1340, a second user input is received that identifies atleast a part of a trajectory of the medical procedure within theinteractive image. In some examples, the trajectory may be identified byconnecting the target to a closest passageway among the connectedanatomical passageways. For example, the second user input received caninclude a line drawn between the target and the closest passageway via aline tool provided by the graphical user interface.

At a process 1350, a third user input is received that adjusts one ormore aspects of the interactive image based at least partly on thedefined trajectory. According to some embodiments, process 1350 maygenerally correspond to method 1200 for growing the anatomical model, inwhich case the third user input may include one or more user inputsreceived during method 1200. For example, process 1350 may includedetermining a distance represented by the trajectory, e.g. a distancebetween the closest passageway and the target. Consistent with suchexamples, adjusting the interactive image may include connecting anunconnected passageway to the connected passageways when the distance isgreater than a predetermined threshold (e.g., 3 cm). The unconnectedpassageway may be connected by receiving a fourth user input identifyingan unconnected passageway that is closer to the target than the nearestpassageway and using automated techniques to connect the identifiedpassageway. In some examples, adjusting the interactive image mayinclude determining an exit angle based on the trajectory (e.g., anangle at which a medical instrument punctures a lumen of the passagewaywhen accessing the target from the passageway) and receiving the thirduser input to manipulate a control provided by the graphical userinterface, such as a slider, to alter the position of an exit pointalong the passageway. In some examples, the control may providecontinuous control over the position of the exit point and/or mayprovide real-time updated metrics associated with the selected exitpoint, such as the corresponding exit angle.

FIGS. 14A-14F are further simplified diagrams of the graphical userinterface 400 in a branch labeling mode according to some embodiments.In this example, the branch labeling mode is applied to lung anatomy,although in further examples, the graphical user interface 400 is usedto label any suitable anatomical structures. Lungs include a right lungand a left lung, where each lung is divided into lobes, which in turncan be divided into segments and lobules. Within each lobe are variousanatomical structures including a set of anatomical passageways 1402which can include a plurality of branches 1403. In the example of FIGS.14A-15E, the branch labeling mode may be used to identify and labelwhich lobe an individual branch 1403 may belong. In alternativeembodiments, the branch labeling mode is used to identify and labelwhich lung, segment, and/or lobule and individual branch 1403 maybelong. In the branch labeling mode, the graphical user interface 400provides a mechanism for a user to assign labels to the branches 1403reflecting a section 1404 of the lung (e.g., lobe) to which therespective branch 1403 belongs. In an exemplary embodiment, the branchlabeling mode is used to aid in registration of an airway model to humananatomy to provide navigation during an image guided biopsy procedure.Further details on registration can be found in 62/486,879, which hasbeen previously incorporated by reference.

Branch labeling may operate on a 3D model, such as that described above,or any other suitable model of the anatomical passageways 1402. Themodel may be created from imaging data including CT data, MM data, OCTdata, x-ray data, and/or other types of imaging or patient data aspreviously described. In some embodiments, the 3D model includes the setof anatomical passageways 1402 and other anatomical structures such asribs, blood vessels, tumors, lesions, and/or organs (e.g. heart). In theillustrated embodiments, the 3D model includes lung branches 1403 andlung pleura 1405, although further embodiments include any type ofsuitable anatomical passageways 1402 and surrounding structure. Thegraphical user interface 400 displays the elements of the 3D model andthe display of individual elements (e.g., the lung pleura 1405) may behidden or displayed to improve clarity.

In some examples, the graphical user interface 400 automates aspects ofthe labeling process including label selection. Furthermore in someexamples, the graphical user interface 400 automates aspects of labelverification including identifying unlabeled branches 1403 and/oridentifying conflicts in user input. In these examples and others, thegraphical user interface 400 may accelerate the planning of a medicalprocedure by providing a user with an expedited process for identifyingbranches.

FIG. 14A illustrates an example of the graphical user interface 400 inthe branch labeling mode prior to labeling the section/lobe 1404 of anyof the branches 1403. In some embodiments, the graphical user interface400 includes an interactive window 1406 for viewing, selecting,labeling, and/or otherwise interacting with the model of the anatomicalpassageways 1402. The interactive window 1406 may display an image ofthe model that represents the anatomical passageways 1402 andsurrounding anatomy (e.g., lung pleura 1405) as 3D renderings,wireframes, cross sections, and/or other suitable representation. In onesuch embodiment, the interactive window 1406 represents the branches1403 by centerlines.

In some embodiments, the graphical user interface 400 includes a labeltool 1408 for selecting labels to assign to branches of the model. Thelabel tool 1408 may be represented as a palette, header, footer,sidebar, menu, message bar, dropdown menu, pop-up window, and/or othersuitable representation. In some embodiments, the label tool 1408displays a list 1410 of labels to be applied. The label tool 1408 mayindicate a currently-selected label using highlighting, color, font,outlining, emphasis, and/or other suitable indicators. The label tool1408 may also display a status indicator 1412 for each label indicatinga status such as whether or not the label has been applied. The statusindicator 1412 may indicate whether a label has been applied to morethan one branch, and in one such embodiment, the status indicator 1412displays a single checkmark to indicate a label has been applied to asingle branch and displays two checkmarks to indicate that the label hasbeen applied to more than one branch. Additionally or in thealternative, the label tool 1408 may display a set of interactiveobjects 1414 (e.g., push buttons, checkboxes, radio buttons, text-basedbuttons, etc.) for setting attributes of the respective label. Forexample, a user may select an interactive object 1414 to indicate thatthe branch to which the label corresponds is missing and not present inthe model and/or the anatomy.

The graphical user interface 400 is operable to receive user input anddisplay a cursor 1416 responsive to the user input. When the cursor 1416is positioned within a boundary of the label tool, the graphical userinterface 400 may receive user input to select a label from the list1410, activate or deactivate an interactive object 1414, and/or takeother suitable action. When the cursor 1416 is positioned within theinteractive window 1406, the graphical user interface 400 may receiveuser input to select a branch of the model, manipulate (e.g., rotate,pan, zoom, etc.) the image of model and/or image data of the surroundinganatomy, and/or take other suitable action. In an example, the graphicaluser interface 400 makes a selection in response to a first mouse buttonwithin the interactive window 1406 and rotates the perspective shown inthe interactive window 90° in response to a second mouse button. Whenrotating the perspective, the graphical user interface 400 may alsorotate a patient orientation indicator 1417 that represents theperspective of the model and surrounding anatomy being displayed in theinteractive window 1406 relative to a patient.

When a label is selected and the cursor 1416 is positioned over a branch1403 in the interactive window 1406, the graphical user interface 400may indicate that the branch 1403 may be labeled by changing arepresentation of the cursor 1416, a representation of the branch 1403or a plurality of branches extending from the selected branch in theinteractive window 1406, and/or providing other suitable indication.Additionally or in the alternative, when a label is selected and thecursor 1416 is positioned in the interactive window 1406 but not overany branch 1403, the graphical user interface 400 may indicate thatcursor is not over a branch by changing a representation of the cursor1416 and/or providing other suitable indication.

Examples of the graphical user interface 400 responding to user inputare described with reference to FIGS. 14B-14F. FIG. 14B illustrates anexample of applying a label to a branch 1403 of the anatomicalpassageways 1402 via the graphical user interface 400. In someembodiments, the label tool 1408 of the graphical user interface 400automatically selects the first label to apply. The label tool 1408 mayselect the first label from those labels in the list 1410 that have notalready been applied to at least one branch. The label tool 1408 mayalso provide an indication, such as those described above, that thefirst label has been selected. The user may override this automaticselection by selecting a different label via the label tool 1408. In theexample illustrated in FIGS. 14A-14F, the labels are indicative of lunglobes. It should be understood that the labels can be indicative ofother sections of the lungs including left or right lung, lung segments,and/or lung lobules.

The graphical user interface 400 may then receive a user selection of abranch 1403 via the interactive window 1406 and, in response, may assignthe selected label to the selected branch 1403. Selecting a singlebranch 1403 may cause the graphical user interface 400 to identify otherbranches 1403 connected to the selected branch and assign the label to aplurality of branches 1403 as a whole. In some examples, a labelindicator 1418 is displayed in the interactive window 1406, such as aflag, a marker, a text box, and/or other suitable indicator thatrepresents the assigned label and the corresponding branch 1403 orplurality of branches 1403. The representation of the branch 1403 orplurality of branches 1403 in the interactive window 1406 may becolored, outlined, emphasized, deemphasized, or otherwise modified toindicate that the label has been assigned. The graphical user interface400 may also update the respective status indicator 1412 in the labeltool 1408 to indicate that the label has been assigned to at least onebranch. In the illustrated example, a first branch 1403 is selected bythe user and labeled as “Right Upper Lobe”. The graphical user interface400 highlights all branches connected to the first branch 1403 up to amain trunk (e.g. trachea of the lung) with a first color. Allhighlighted branches are effectively identified and labeled as belongingto the “Right Upper Lobe” section/lobe 1404 at this point.

As depicted in FIG. 14C, when a label is applied to a branch 1403 or aplurality of branches 1403, the label tool 1408 may automatically selectthe next label to be applied in the list 1410. In some embodiments, thelabel tool 1408 selects the next label based on the arrangement of thebranches in the anatomical passageways 1402. For example, a middle lobelabel may be selected after assigning an upper lobe label because theupper lobe is proximate to the middle lobe in the anatomical passageways1402. In some embodiments, the label tool 1408 selects from those labelsin the list 1410 that have not yet been applied to any branch. Forexample, a lower lobe label may be selected after assigning an upperlobe label if the middle lobe label has already been used to label abranch. By automatically selecting the next label, in these embodimentsand others, the user may continue selecting branches without moving thecursor out of the interactive window 1406.

The process of selecting labels and branches may be repeated. Asexplained above in the context of FIG. 14B, in an example, a firstbranch 1403 is selected by the user and labeled as “Right Upper Lobe”.As a result, all branches connected to the first branch 1403 up to amain trunk are also labeled as “Right Upper Lobe.” In response, thegraphical user interface 400 selects the next label, “Right MiddleLobe,” as shown in FIG. 14C. Referring to FIG. 14D, a user selects asecond branch 1403 to assign the label “Right Middle Lobe.” The graphicuser interface 400 assigns the label to and alters the color of allbranches 1403 that are descendants of the second branch (e.g. childbranches that stem from the second branch or are distal to and connectedto the second branch) with a second color, identifying and labeling thedescendant branches as belonging to a right middle lobe. In one example,when the second branch 1403 is selected, all descendant branches arehighlighted with the second color and a portion of proximal/parentconnected branches reflect a change in color up to a main trunk of thepreviously identified section/lobe 1404. In one embodiment, a pluralityof branches 1403 may overlap, potentially belonging to two separatesections/lobes 1404. The overlapping branches can be highlighted in aseparate specific color reflecting the overlap. This process is repeatedfor a “Right Lower Lobe” using a third color to identify and labelbranches distal to a third user selected branch within the right lowerlobe. The process can again be repeated for a “Left Upper Lobe” and“Left Lower Lobe” until all the branches 1403 have been identified andlabeled as belonging to a section/lobe 1404 as shown in FIG. 14E.

As depicted in FIG. 14E, to assist the user, in some embodiments, thegraphical user interface 400 indicates in the interactive window 1406those branches 1403 in the anatomical passageways 1402 that do not yethave a label. The graphical user interface 400 may use indicators 1420such as flags, highlighting, outlines, color, line weight, and/or othersuitable indicator to indicate unlabeled branches 1403. In one suchexample, the graphical user interface 400 displays a question mark flagwith a connector extending to an unlabeled branch 1403. In someembodiments, the graphical user interface 400 indicates the remainingunlabeled branches for the user's reference when it detects that eachlabel has been applied to at least one branch 1403. When the userselects a label and selects an unlabeled branch, the label may beapplied to the branch 1403 and/or the section/lobe 1404 to which itbelongs, and the indicator 1420 may be removed, as shown in FIG. 14F. Inexamples where unlabeled branch detection is performed when each labelhas been applied to at least one branch, applying a label to anunlabeled branch may result in the label being applied to more than onebranch. As a result, the graphical user interface 400 may update thecorresponding status indicator accordingly. In the examples of FIG. 14E,status indicator 1412A is updated to display a pair of checkmarksbecause the left upper lobe has been assigned to more than onesection/lobe 1404. In a further example where unlabeled branch detectionis performed when all labels have been applied to at least one branch,the graphical user interface 400 removes the unlabeled branch indicator1420 when a label is deleted from a branch and it is no longer the casethat all labels have been applied to at least one branch.

FIGS. 15A and 15B are simplified diagrams of a method 1500 for planninga medical procedure according to some embodiments. According to someembodiments consistent with FIGS. 14A-14F, method 1500 may be used tooperate a graphical user interface, such as graphical user interface400, in a branch labeling mode.

At process 1501 of FIG. 15A, imaging data such as CT data of a patientis received. At process 1502, a model of the patient anatomy is renderedfrom the image data. The model may include a set of anatomicalpassageways 1402 and/or other anatomical structures such as organs,blood vessels, tumors, lesions, etc. In particular, the anatomicalpassageways 1402 may include branches 1403 and/or other suitablestructure.

At process 1504, an image of the model is displayed via the graphicaluser interface 400. According to some embodiments, the image of themodel is displayed in an interactive window 1406 that includesrepresentations of the anatomical passageways 1402 and of thesurrounding anatomy (e.g., lung pleura 1405). These elements of thesurrounding anatomy may be individually displayed or hidden to provide aframe of reference and improve clarity.

At process 1506, a first label is selected. The first label may beselected automatically and/or in response to a first user input receivedby a label tool 1408 of the graphical user interface. The user input maybe provided via any suitable input mechanism including a mouse,keyboard, touch, stylus, trackball, joystick, speech commands, virtualreality interface, and/or the like. At process 1508, an indication thatthe first label has been selected is displayed by a label tool 1408 ofthe graphical user interface.

At process 1510, the graphical user interface 400 may receive a seconduser input selecting a first branch 1403 for the first label. The seconduser input may select the first branch 1403 via the interactive window1406 in which the model is displayed. At process 1511, in some examples,the graphical user interface 400 identifies other branches 1403connected to the first branch 1403 so that a plurality of branches 1403may be labeled in a single process. In an example, this includesidentifying those branches 1403 that are descendants of the first branch1403 (e.g., child branches that stem distally from the first branch) andincluding the descendants in the section/lobe 1404. In this way, a labelmay propagate downstream from the selected branch. In an example, thisincludes identifying antecedent branches 1403 (e.g., parents,grandparents, etc. that stem proximally from the first branch) up to amain branch. The main branch can be the trachea of the lung in initialcases or a main branch of a previously identified section/lobe 1404. Inthis way, a label can propagate upstream so that a user may label asubtree without necessarily selecting the root of the subtree.

At process 1512, the graphical user interface 400 determines whether thelabel conflicts with a previously selected label. For example, a usermay label a child branch with a right upper lobe label. If the childbranch includes a parent branch that was previously labeled with a leftupper lobe label, the algorithm can recognize that the conflict betweenthe child and the parent branch. The graphical user interface 400 mayreject applying the current label and illustrate the conflict byhighlighting the current label, the currently-selected branch, theprevious label, and/or the previously-selected branch. The user may thenbe presented the option to correct the previous label or the currentlabel. Once any conflict is resolved, at process 1513, the first branch1403 and any other branches 1403 identified in process 1511 are labeledwith the first label.

At process 1514, the graphical user interface displays a representationof the first label applied to the first branch 1403 and/or itsrespective section/lobe 1404. In some such examples, the graphical userinterface displays a flag, a marker, a text box, and/or other suitableindicator in the interactive window 1406 to represent the first label.In some examples, the representation of the first branch 1403 and/or itssection/lobe 1404 in the interactive window 1406 is colored, outlined,emphasized, deemphasized, or otherwise modified to indicate that thelabel has been assigned.

Referring now to FIG. 15B, at process 1516, it is determined whether thelabel tool 1408 has additional unassigned labels. If so, at process1518, a next label is selected. The next label may be selectedautomatically from a set of unassigned labels based on the arrangementof the branches 1403 in the anatomical passageways 1402 and/or othersuitable criteria. The next label may also be selected in response to afirst user input received by the label tool 1408 of the graphical userinterface. In some such examples, the user selection overrides theautomatic selection. At process 1520, an indication that the first labelhas been selected is displayed by the label tool 1408 of the graphicaluser interface. At process 1522, the graphical user interface mayreceive a user input selecting a branch of the anatomical passagewaysfor the label. In response, at process 1524, the selected branch islabeled with the label. At process 1526, the graphical user interfacedisplays a representation of the label applied to the branch. Processes1518-1526 may be performed substantially similar to processes 1506-1514.

At process 1528, the graphical user interface may identify a branch 1403of the anatomical passageways 1402 that does not have an assigned label.This may be performed when it is determined that each label has beenassigned to at least one branch in process 1516. At process 1530, thegraphical user interface displays an indicator that identifies theunassigned branch in the interactive window 1406 such as a flag, ahighlight, an outline, a color, line weight, and/or other suitableindicator. At process 1532, a user input is received selecting a label.This may be performed substantially similar to process 1506 and/or 1518.At process 1534, the selected label is applied to the unassigned branchand its respective section/lobe, and at process 1536, the graphical userinterface displays a representation of the label applied to the branchand/or section/lobe. Processes 1534-1536 may be performed substantiallysimilar to processes 1511-1514. This may be repeated until each branchis assigned a label.

FIG. 16 is a simplified diagram of a method 1600 for planning a medicalprocedure according to some embodiments. FIGS. 17A-P are correspondingdiagrams of graphical user interface 400 during the performance ofmethod 1600 according to some embodiments. According to some embodimentsconsistent with FIGS. 1-15B, graphical user interface 400 may includefeatures that generally correspond to similar features depicted in FIGS.1-15B, such as interactive header 410, control frame 510, canvas frame520, workspace 522, tool selector 524, interactive window 541, selectionsidebar 542, image data 543, segmentation data 544, and/or the like. Insome embodiments, graphical user interface 400 may include a modeselector 1712 (illustratively placed within interactive header 410)and/or a view selector 1714 (illustratively placed at the lower sectionof interactive window 541) for enabling and/or disabling variousfeatures of graphical user interface 400, including various featuresused during performance of method 1600.

At a process 1610, a target 1720 of the medical procedure is added viagraphical user interface 400. Illustrative screenshots corresponding toprocess 1610 are depicted in FIGS. 17A and 17B. In some embodiments,target 1720 may generally correspond to target 550. As depicted in FIG.17A, during addition of target 1720, a new target menu 1722 may bedisplayed in control frame 510. New target menu 1722 may displayinstructions for identifying, placing, adjusting, and/or otherwiseconfiguring target 1720, controls for confirming and/or cancelling theaddition of target 1720, and/or the like. When target 1720 has beenadded, target data 1724 may be displayed in control frame 510. Asdepicted in FIG. 17B, target data 1724 may include metrics associatedwith target 1720 (e.g., size metrics), controls (e.g., controls todelete, rename, and/or edit target 1720), and/or the like.

At a process 1620, an operator may optionally zoom to target 1720(and/or other portions of image data 543) view graphical user interface400. Illustrative screenshots corresponding to process 1610 are depictedin FIGS. 17C-17E. Graphical user interface 400 may provide one or moreoptions for zooming to target 1720, including a non-synchronized zoom, asynchronized zoom, an auto-zoom, and/or the like. FIG. 17C depicts azoom performed when a synchronized zoom feature of graphical userinterface 400 is not enabled, and FIG. 17D depicts a zoom performed whenthe synchronized zoom feature is enabled. When the synchronized zoomfeature is not enabled, zooming in and/or out of the image displayed ininteractive window 541 is not accompanied by a matching zoom effect inthe thumbnail images displayed in selection sidebar 542. Conversely,when the synchronized zoom feature is enabled, zooming in and/or out ofthe image displayed in interactive window 541 is accompanied by amatching zoom effect in the thumbnail images displayed in selectionsidebar 542. FIG. 17E depicts a zoom performed using an auto-zoomfeature of graphical user interface 400. Using the auto-zoom featurescauses target 1720 to be automatically centered and/or magnified in theimage displayed in interactive window 541. The auto-zoom feature may ormay not be accompanied by a matching zooming effect in the thumbnailimages displayed in selection sidebar 542.

At a process 1630, an operator may optionally edit target 1720. Anillustrative screenshot corresponding to process 1630 is depicted inFIG. 17F. As depicted in FIG. 17F, during editing of target 1720, anedit target menu 1726 may be displayed in control frame 510. Edit targetmenu 1726 may display instructions for editing target 1720, controls forconfirming and/or cancelling the editing of target 1720, and/or thelike. In some embodiments, the operator may modify attributes of target1720 (e.g., the size, position, and/or the like) via interactive window541 and/or via selection sidebar 542.

At a process 1640, a path 1730 to target 1720 is identified viagraphical user interface 400. Illustrative screenshots corresponding toprocess 1640 are depicted in FIGS. 17G-171. As depicted in FIG. 17G,path data 1732 corresponding to a selected path may be displayed incontrol frame 510. Path data 1732 may include metrics associated withpath 1730 (e.g., distance between the endpoint of the path and target1720, exit angle at the endpoint of the path, and/or the like), controls(e.g., controls to delete, rename, and/or edit path 1730), and/or thelike. In some embodiments, one or more alerts associated with the pathmay be displayed via graphical user interface 400. For example, asdepicted in FIG. 17G, an alert 1734 is displayed when the distancebetween the endpoint of the path and the nearest point of target 1720exceeds a predetermined threshold. Similarly, as depicted in FIG. 17H,an alert 1736 is displayed when the exit angle at the endpoint of thepath exceeds a predetermined threshold. In some embodiments, an endpointslider 1738 may be displayed in control frame 510 to allow the operatorto adjust the position of the endpoint along the path. In this regard,the operator may determine a position of the endpoint that is incompliance with the predetermined thresholds associated with thedistance to target 1720 and/or the exit angle.

At a process 1650, one or more passageways are optionally extended viagraphical user interface 400. For example, the one or more passagewaysmay be extended when an acceptable path to target 1720 is not identifiedat process 1640. Illustrative screenshots corresponding to process 1650are depicted in FIGS. 17J-17M. As depicted in FIG. 17J, an instructionpanel 1740 may be displayed in control frame 510 to provide instructionsfor extending the one or more passageways. As depicted in FIG. 17K, theoperator may draw a path extension 1742 via interactive window 541. Pathextension 1742 may subsequently be rendered, as depicted in FIG. 17L. Insome embodiments, path extension 1742 may be rendered in a differentcolor, texture, pattern, and/or the like to distinguish path extension1742 from portions of the path determined by segmentation. As depictedin FIG. 17M, the updated path to target 1720 may be in compliance withthe predetermined thresholds associated with the distance to target 1720and/or the exit angle, such that alerts 1734 and/or 1736 are no longerdisplayed with path data 1732.

At a process 1660, the plan for the medical procedure is reviewed viagraphical user interface 400. An illustrative screenshot correspondingto process 1660 is depicted in FIG. 17N. The features depicted in FIG.17N generally correspond to those of the preview mode of graphical userinterface 400 as depicted in FIG. 6. Consistent with such embodiments,FIG. 17N depicts a virtual endoscopic view 620, a global anatomicalmodel view 630, and a reduced anatomical model view 640. As depicted inFIG. 17N, global anatomical model view 630 includes controls, e.g., toallow the operator to pan, zoom, and/or rotate the model view and/orselect among different types of images (e.g., CT images). Likewise,reduced anatomical model view 640 includes controls to start and/orpause playback of the plan for the medical procedure, to export thecurrent plan, and/or the like. In some embodiments, an information panel1750 may be displayed to provide information associated with the plan,such as metrics associated with the distance to target 1720.

Some examples of control units, such as control unit 130 may includenon-transient, tangible, machine readable media that include executablecode that when run by one or more processors (e.g., processor 140) maycause the one or more processors to perform the processes of methods1000-1300 and/or method 1500 to render graphical user interface 400.Some common forms of machine readable media that may include theprocesses of methods 1000-1300, method 1500, and/or the instructions forrendering graphical user interface 400 are, for example, floppy disk,flexible disk, hard disk, magnetic tape, any other magnetic medium,CD-ROM, any other optical medium, punch cards, paper tape, any otherphysical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM,any other memory chip or cartridge, and/or any other medium from which aprocessor or computer is adapted to read.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. Thus, the scope of theinvention should be limited only by the following claims, and it isappropriate that the claims be construed broadly and in a mannerconsistent with the scope of the embodiments disclosed herein.

1. A planning workstation comprising: a display system; and a user inputdevice; wherein the planning workstation is configured to: display imagedata via the display system; receive a first user input via the userinput device; display via the display system a target of a medicalprocedure within the displayed image data identified based at least onthe first user input; display an interactive image via the displaysystem, the interactive image comprising the image data, a plurality ofconnected anatomical passageways, and the identified target; receive asecond user input via the user input device; display via the displaysystem a trajectory between the target and an exit point along a nearestpassageway of the plurality of connected anatomical passagewaysidentified based at least on the second user input; receive a third userinput via the user input device; and adjust the interactive image basedat least on the trajectory and the third user input.
 2. The planningworkstation of claim 1, wherein the user input device includes atouchscreen of the display system.
 3. The planning workstation of claim1, wherein the display system adjusts the interactive image including:determine a distance represented by the trajectory; determine whetherthe distance is greater than a predetermined threshold; receive a fourthuser input via the user input; identify an unconnected passageway thatis closer to the target than the nearest connected passageway based atleast on the fourth user input; and connect the unconnected passagewayto the plurality of connected passageways.
 4. The planning workstationof claim 3, wherein the planning workstation is further configured toreceive a fifth user input via the user input device, and rotate theinteractive image to identify the unconnected passageway in theinteractive image based at least on the fifth user input.
 5. Theplanning workstation of claim 4, wherein the interactive image isrotated about one or more user-defined rotation points.
 6. The planningworkstation of claim 1, wherein the display system adjusts theinteractive image including: determine an exit angle based on thetrajectory; and adjust the exit angle by altering a position of the exitpoint along the nearest connected passageway.
 7. The planningworkstation of claim 1, wherein the planning workstation is furtherconfigured to receive a fourth user input via the user input device, anddisplay via the display system a hazard of the medical procedure withinthe displayed image data based on the fourth user input.
 8. The planningworkstation of claim 7, wherein the hazard corresponds to at least oneof a vulnerable portion of a patient anatomy and an excessive bend inone or more of the plurality of connected anatomical passageways.
 9. Theplanning workstation of claim 7, wherein the hazard is displayed using ahazard fence that includes at least one of a circular disk, a conichazard fence, or a hemispherical hazard fence.
 10. A method of planninga medical procedure, the method comprising: receiving a representationof anatomical passageways including a plurality of branches; displaying,via a graphical user interface, an image of the representation;receiving a first user input representing selection of a first label;receiving a second user input representing selection of a first branchof the plurality of branches; and in response to the first user inputand the second user input: labeling the first branch with the firstlabel; and displaying, via the graphical user interface, arepresentation of the first label applied to the first branch.
 11. Themethod of claim 10 further comprising, based on the labeling of thefirst branch with the first label: selecting a second label; anddisplaying, via the graphical user interface, an indication that thesecond label has been selected.
 12. The method of claim 11, wherein thesecond label is selected based on an arrangement of the plurality ofbranches within the anatomical passageways.
 13. The method of claim 11further comprising: receiving a third user input representing selectionof a second branch of the plurality of branches; labeling the secondbranch with the second label; and displaying, via the graphical userinterface, a representation of the second label applied to the secondbranch.
 14. The method of claim 10 further comprising: identifying agroup of branches from the plurality of branches that includes the firstbranch; and in response to the first user input and the second userinput, labeling the group of branches with the first label.
 15. Themethod of claim 14 wherein the identifying of the group of branchesincludes identifying a descendent branch of the first branch andincluding the descendent branch in the group of branches.
 16. The methodof claim 10 further comprising, displaying via the graphical userinterface, an indication of a second branch of the plurality of branchesthat does not have an assigned label.
 17. The method of claim 10 furthercomprising, in response to the first user input and the second userinput: assigning a color to the first branch; and displaying the imageof the representation of anatomical passageways with the first branchcolored with the assigned color.
 18. The method of claim 10 furthercomprising: providing, via the graphical user interface, a cursor;detecting that the cursor is aligned with the first branch of theplurality of branches; and based on detecting that the cursor is alignedwith the first branch, modifying a representation of the cursor.
 19. Themethod of claim 10 further comprising: receiving a third user inputrepresenting a rotation instruction; in response to the third userinput, rotating the representation of anatomical passageways; anddisplaying, via the graphical user interface, an image of the rotatedrepresentation of anatomical passageways.
 20. A non-transitory machinereadable medium comprising a plurality of machine readable instructionswhich when executed by one or more processors associated with a planningworkstation are adapted to cause the one or more processors to perform amethod comprising: providing a plurality of interactive windows for auser to view a plan for a medical procedure, wherein each of theplurality of interactive windows displays a different rendering of amodel of anatomical passageways; displaying a path through theanatomical passageways to a target of the medical procedure; displayinga virtual image of an instrument within the anatomical passageways;displaying a control point corresponding to a distal end of theinstrument in at least one of the plurality of interactive windows;receiving a user input; identifying a position of the control pointbased at least on the user input; and in response to receiving the userinput, dynamically updating a position of the instrument in at least oneof the plurality of interactive windows to match the position of thecontrol point.